· Xiaojing Yang · Statistics · 3 min read
中文Hypothesis Testing Without Rituals
Hypothesis testing is a way to discipline claims, not a ritual for producing p-values.
Core idea
A test is useful only when the hypothesis, data, metric, and claim actually match.
1. What hypothesis testing is for
The point of hypothesis testing is not to decorate a paper with p-values. It is to ask whether the observed evidence is surprising under a baseline assumption.
No real difference or no effect
What the experiment produced
A summary of the evidence
What would happen under the null
How cautious the claim should be
2. The p-value trap
A small p-value does not mean the effect is large. It also does not mean the hypothesis is true. It means that, under the null model, results at least this extreme would be unusual.
This distinction matters in AI because large benchmarks can make tiny effects look statistically significant while small benchmarks can hide useful effects.
3. Match the test to the experiment
| Situation | Better question |
|---|---|
| Same examples evaluated by two models | Use a paired comparison |
| Many prompts tested | Control multiple comparisons |
| Non-normal metric distribution | Prefer bootstrap or permutation logic |
| Human labels with disagreement | Model uncertainty in annotation |
4. AI/NLP example
In machine translation evaluation, a paired bootstrap test asks whether the observed BLEU difference remains under resampling. In LLM evaluation, a permutation or paired test can ask whether one model wins consistently on the same items.
Bad use
Run a test after looking at many metrics, then report the smallest p-value.
Good use
Define the comparison first, choose the test for the data structure, and report uncertainty plus effect size.
Takeaway
Hypothesis testing should make research claims more humble and precise. If it becomes a ritual, it can make weak claims look stronger than they are.
References and learning path
This note uses the statistics-to-machine-learning route that fits my AI/NLP research goals: build intuition with Seeing Theory and StatQuest, connect it to Python practice with Think Stats, then deepen the ML connection with ISLR/ISLP, CS229, and selected statistical inference references.