type
status
date
slug
summary
tags
category
icon
password
What is A/B Testing?
A/B testing is an experimental method used to compare two versions of a product or feature to determine which one performs better based on a predefined metric. It is widely used in product development, marketing, UI/UX design, and data-driven decision making.
Key Concepts
- Control Group (A): The current version, used as the baseline.
- Variant Group (B): A modified version to be tested against the control.
- Metric: The performance indicator (e.g., click-through rate, conversion rate, time on page).
- Randomization: Users are randomly assigned to either group to ensure unbiased comparisons.
Example Scenario
An e-commerce site wants to test if changing the "Buy" button color improves conversion rate.
Group | Users | Purchases | Conversion Rate |
A | 5000 | 250 | 5.0% |
B | 5000 | 300 | 6.0% |
Group B appears to have a higher conversion rate. However, we must determine whether this difference is statistically significant or due to random variation.
Hypothesis Testing Framework
- Null Hypothesis (H₀): Conversion rates in Group A and B are equal.
- Alternative Hypothesis (H₁): Conversion rates differ.
We use a two-proportion z-test to evaluate the difference.
Step-by-Step Statistical Test
- Observed proportions
p1=250/5000=0.05
p2=300/5000=0.06
- Pooled proportion
- Standard error (SE)
- Z-score
- P-value
For z = 2.22, p-value ≈ 0.026
- Decision rule
If p-value < 0.05, reject the null hypothesis.
Since 0.026 < 0.05, the difference is statistically significant.
Group B (red button) has a significantly higher conversion rate than Group A (blue button). Therefore, we conclude that the red button performs better and is likely to lead to more purchases.
Practical Notes
- Only one variable should change between A and B.
- Ensure sufficient sample size for statistical power.
- Use random assignment to avoid bias.
- Don't stop the test early based on initial trends—this increases the risk of Type I errors.
- Author:Entropyobserver
- URL:https://tangly1024.com/article/1f3d698f-3512-80fb-9fa9-def691c20b74
- Copyright:All articles in this blog, except for special statements, adopt BY-NC-SA agreement. Please indicate the source!