· Xiaojing Yang · Statistics · 2 min read
中文Multiple Comparisons in AI Experiments
When we try many models, prompts, seeds, and metrics, false discoveries become easier than they look.
Core idea
The more comparisons we try, the easier it is to find a “winner” by chance.
1. The hidden multiplication
AI experiments often contain many comparisons: models, checkpoints, prompts, datasets, random seeds, metrics, and subgroups. Even if each test has a low false-positive rate, the whole experiment can accumulate many chances to fool us.
A, B, C...
templates and wording
training randomness
accuracy, F1, COMET...
domains, languages, user groups
2. The research risk
If we report only the best result after many trials, the result may reflect search over randomness rather than a stable improvement.
Visible report
One clean winning number.
Hidden process
Many attempted comparisons behind the winner.
3. AI/NLP example
Prompt engineering is especially vulnerable. If I test 40 prompt variants and report the best one without validation, the selected prompt may simply fit quirks of the development set.
4. Better habits
| Habit | Why it helps |
|---|---|
| Separate dev and test | Prevent final test from becoming a tuning set |
| Report search space | Shows how many chances existed |
| Use correction when needed | Controls false positives |
| Validate the selected method | Checks whether the winner generalizes |
Takeaway
Multiple comparisons are not a minor statistical technicality. They are one of the main ways AI experiments accidentally overclaim.
References and learning path
This note uses the statistics-to-machine-learning route that fits my AI/NLP research goals: build intuition with Seeing Theory and StatQuest, connect it to Python practice with Think Stats, then deepen the ML connection with ISLR/ISLP, CS229, and selected statistical inference references.