· Xiaojing Yang · Statistics · 2 min read
中文Statistical Testing for NLP and LLM Evaluation
A practical map for choosing statistical tests in NLP, MT, RAG, and LLM evaluation.
Core idea
The right test depends less on the metric name and more on the structure of the comparison.
1. Start from the comparison
Before choosing a statistical test, describe the experiment. Are two systems evaluated on the same items? Are labels binary, ordinal, continuous, or human-ranked? Are there multiple languages or domains?
paired or unpaired
binary, continuous, ranking
correction needed
annotation uncertainty
what the test supports
2. Common NLP settings
| Setting | Useful approach |
|---|---|
| MT metric comparison | paired bootstrap or approximate randomization |
| Classification accuracy | McNemar-style paired logic or bootstrap |
| F1 / macro metrics | bootstrap over examples |
| Human preference | paired tests or hierarchical models |
| Many prompts/models | multiple-comparison control |
3. LLM evaluation complications
LLM evaluation adds prompt sensitivity, stochastic decoding, judge models, contamination risk, and item difficulty. A statistical test cannot remove these design issues; it only helps quantify uncertainty under the chosen protocol.
Test can answer
Is the observed difference stable under this protocol?
Test cannot answer alone
Is the benchmark valid, unbiased, or sufficient for the real task?
4. Reporting pattern
I would report:
- metric scores;
- uncertainty intervals;
- paired comparison result;
- effect size;
- error analysis by category;
- limitations of the benchmark and judge.
Takeaway
Good NLP/LLM evaluation is not one test. It is a chain of evidence: metric, uncertainty, paired comparison, effect size, and qualitative error analysis.
References and learning path
This note uses the statistics-to-machine-learning route that fits my AI/NLP research goals: build intuition with Seeing Theory and StatQuest, connect it to Python practice with Think Stats, then deepen the ML connection with ISLR/ISLP, CS229, and selected statistical inference references.