· Xiaojing Yang · Machine Learning · 2 min read
中文Model Selection: A Practical Guide
Model selection is the disciplined process of choosing among models without fooling yourself.
Core idea
Model selection is not picking the highest number; it is choosing the model with the best evidence for the real task.
1. What model selection includes
Model selection includes choosing algorithms, features, preprocessing, hyperparameters, thresholds, and sometimes the metric itself. The danger is that every choice can overfit the validation set.
Define options
Choose fair comparison
Match task cost
Pick using validation evidence
Estimate once on untouched data
2. Practical criteria
| Criterion | Why it matters |
|---|---|
| Performance | Does it solve the task? |
| Stability | Does it survive different splits/seeds? |
| Simplicity | Is the complexity justified? |
| Cost | Training and inference budget |
| Interpretability | Can errors be explained? |
| Robustness | Does it hold across domains? |
3. sklearn example
from sklearn.model_selection import cross_validate
results = cross_validate(model, X, y, cv=5, scoring=["accuracy", "f1_macro"], return_train_score=True)4. AI/NLP connection
For NLP, the best average score may not be the best model. A model that performs slightly worse overall but handles rare domain terminology, minority languages, or severe-error cases better may be the stronger research choice.
Leaderboard thinking
One metric decides everything.
Research thinking
Evidence, uncertainty, cost, and failure modes decide together.
Takeaway
Model selection is a research judgment process, not a single sorting operation.
Interview pattern
When this appears in an interview, I would answer in four layers:
- give the short definition;
- explain the intuition;
- name the common failure mode;
- connect it to a real evaluation or deployment decision.