· Xiaojing Yang · Machine Learning · 2 min read
中文Overfitting and Regularization
How models learn noise, how validation curves reveal it, and how regularization controls it.
Core idea
Overfitting happens when a model becomes excellent at the training sample and unreliable outside it.
1. The intuition
A model should learn reusable structure. Overfitting means it also learns accidental details of the training set: noise, duplicates, annotation quirks, and dataset-specific shortcuts.
Train and validation error both high
Validation improves
Train improves but validation worsens
2. Regularization
Regularization discourages unnecessary complexity. It can appear as L1/L2 penalties, early stopping, dropout, data augmentation, pruning, or architectural constraints.
| Method | Practical effect |
|---|---|
| L2 / weight decay | Keeps weights smaller and smoother |
| L1 | Encourages sparse features |
| Early stopping | Stops before memorization deepens |
| Dropout | Reduces reliance on one path |
| Data augmentation | Makes shortcuts less useful |
3. sklearn example
from sklearn.linear_model import LogisticRegression
model = LogisticRegression(C=0.1, penalty="l2", max_iter=1000)
model.fit(X_train, y_train)In scikit-learn, smaller C means stronger regularization for many linear models.
4. AI/NLP connection
In small-domain NLP datasets, overfitting can mean memorizing document templates or terms that appear in both train and validation. For fine-tuning, regularization also means limiting how much a pretrained model changes.
Interview answer
Overfitting is a generalization failure caused by learning noise or sample-specific patterns.
Research answer
We diagnose it with held-out data, learning curves, seed variation, and domain-specific error analysis.
Takeaway
Regularization is not only a mathematical penalty. It is a way to make the model earn complexity.
Interview pattern
When this appears in an interview, I would answer in four layers:
- give the short definition;
- explain the intuition;
- name the common failure mode;
- connect it to a real evaluation or deployment decision.