A/B test
Showing two versions to randomly split users and measuring which one moves a chosen metric.
Random assignment makes the groups comparable, so the difference in the metric can be credited to the change. Results need enough traffic: with a few hundred visitors a week, only large effects show up, and checking every day until something looks significant produces false winners. Feature flags usually do the splitting.
Related terms
- Feature flags: Switches in code that turn a feature on or off for some or all users without a new deploy.
- Conversion funnel: The ordered steps users take towards a goal, such as visit, sign-up and payment, with the share who get from each step to the next.
- Activation rate: The share of new users who reach the first real use of a product within a set time.
