Definition
An A/B test compares two versions of a page, message or feature by showing each to a randomly chosen group of users and measuring a chosen outcome, such as sign-ups or purchases. Random assignment means any difference in the outcome is caused by the change, not by who saw it.
How it works
- Pick one change and one primary metric, and decide the minimum difference worth acting on.
- Split users at random into A (the current version) and B (the change).
- Run the test until you have enough users to tell a real difference from noise.
- Compare the metric between the groups, and check a counter-metric didn't get worse.
Common mistakes
- Stopping early the moment B looks ahead. Early results swing widely.
- Testing too many things at once, so you can't tell what caused the result.
- Optimising a step, not the goal: more clicks on a button that lead to fewer paid customers.
Measuring the outcome
Send the variant a user saw as a property on your events (variant: "B"), and compare conversion between variants in a funnel or chart. LogZuno doesn't assign variants itself; use your feature-flag or testing tool for that, and LogZuno to measure what users did afterwards.