Lazy loaded image
Mathematics
Lazy loaded imageA/B Testing
Words 291Read Time 1 min
May 14, 2020
May 14, 2025
type
status
date
slug
summary
tags
category
icon
password

What is A/B Testing?

A/B testing is an experimental method used to compare two versions of a product or feature to determine which one performs better based on a predefined metric. It is widely used in product development, marketing, UI/UX design, and data-driven decision making.
Key Concepts
  • Control Group (A): The current version, used as the baseline.
  • Variant Group (B): A modified version to be tested against the control.
  • Metric: The performance indicator (e.g., click-through rate, conversion rate, time on page).
  • Randomization: Users are randomly assigned to either group to ensure unbiased comparisons.
Example Scenario
An e-commerce site wants to test if changing the "Buy" button color improves conversion rate.
Group
Users
Purchases
Conversion Rate
A
5000
250
5.0%
B
5000
300
6.0%
Group B appears to have a higher conversion rate. However, we must determine whether this difference is statistically significant or due to random variation.
Hypothesis Testing Framework
  • Null Hypothesis (H₀): Conversion rates in Group A and B are equal.
  • Alternative Hypothesis (H₁): Conversion rates differ.
We use a two-proportion z-test to evaluate the difference.
Step-by-Step Statistical Test
  1. Observed proportions
    1. p1=250/5000=0.05
      p2=300/5000=0.06
  1. Pooled proportion
  1. Standard error (SE)
  1. Z-score
  1. P-value
    1. For z = 2.22, p-value ≈ 0.026
  1. Decision rule
    1. If p-value < 0.05, reject the null hypothesis.
Since 0.026 < 0.05, the difference is statistically significant.
Group B (red button) has a significantly higher conversion rate than Group A (blue button). Therefore, we conclude that the red button performs better and is likely to lead to more purchases.
Practical Notes
  • Only one variable should change between A and B.
  • Ensure sufficient sample size for statistical power.
  • Use random assignment to avoid bias.
  • Don't stop the test early based on initial trends—this increases the risk of Type I errors.
 
上一篇
Empirical Cumulative Distribution Function
下一篇
TF-IDF Calulation