A/B Testing | EntropyObserver

type

status

date

slug

summary

What is A/B Testing?

A/B testing is an experimental method used to compare two versions of a product or feature to determine which one performs better based on a predefined metric. It is widely used in product development, marketing, UI/UX design, and data-driven decision making.

Key Concepts

Control Group (A): The current version, used as the baseline.

Variant Group (B): A modified version to be tested against the control.

Metric: The performance indicator (e.g., click-through rate, conversion rate, time on page).

Randomization: Users are randomly assigned to either group to ensure unbiased comparisons.

Example Scenario

An e-commerce site wants to test if changing the "Buy" button color improves conversion rate.

Group	Users	Purchases	Conversion Rate
A	5000	250	5.0%
B	5000	300	6.0%

Group B appears to have a higher conversion rate. However, we must determine whether this difference is statistically significant or due to random variation.

Hypothesis Testing Framework

Null Hypothesis (H₀): Conversion rates in Group A and B are equal.

Alternative Hypothesis (H₁): Conversion rates differ.

We use a two-proportion z-test to evaluate the difference.

Step-by-Step Statistical Test

Observed proportions

p1=250/5000=0.05

p2=300/5000=0.06

Pooled proportion

Standard error (SE)

Z-score

P-value

For z = 2.22, p-value ≈ 0.026

Decision rule

If p-value < 0.05, reject the null hypothesis.

Since 0.026 < 0.05, the difference is statistically significant.

Group B (red button) has a significantly higher conversion rate than Group A (blue button). Therefore, we conclude that the red button performs better and is likely to lead to more purchases.

Practical Notes