· Xiaojing Yang · Statistics · 2 min read
中文Bias-Variance Trade-off
Bias and variance explain why both too-simple and too-flexible models can fail.
Core idea
Generalization fails when a model is too rigid to learn the signal or too flexible to ignore the noise.
1. The picture
Bias is error from overly simple assumptions. Variance is error from being too sensitive to the training sample. The trade-off is not just textbook vocabulary; it is a way to diagnose ML behavior.
High bias, underfitting
Signal learned, noise controlled
High variance, overfitting
2. What it looks like
| Symptom | Likely issue |
|---|---|
| Poor train and test performance | High bias |
| Excellent train, poor test | High variance |
| Big seed-to-seed changes | High variance |
| Consistent but mediocre predictions | High bias |
High bias
The model cannot represent the pattern well enough.
High variance
The model changes too much when the data changes.
3. AI/NLP example
In low-resource domain adaptation, a small adapter may underfit domain terminology. A fully fine-tuned large model may overfit a small noisy corpus. The best method is often a compromise: enough capacity to adapt, enough constraint to remain stable.
4. How to respond
High bias suggests better features, larger models, more expressive architectures, or task reformulation. High variance suggests more data, stronger regularization, better validation, ensembling, or simpler adaptation.
Takeaway
Bias-variance thinking gives me a diagnostic map: not just “the model is bad,” but “what kind of bad is it?”
References and learning path
This note uses the statistics-to-machine-learning route that fits my AI/NLP research goals: build intuition with Seeing Theory and StatQuest, connect it to Python practice with Think Stats, then deepen the ML connection with ISLR/ISLP, CS229, and selected statistical inference references.