· Xiaojing Yang · Statistics · 2 min read
中文PCA: From Variance to Representation
PCA explains how variance, projection, and representation are connected.
Core idea
PCA finds directions where the data varies the most, then uses those directions as a simpler representation.
1. The geometric idea
PCA starts with a cloud of points and asks: along which directions does the data vary most? Those directions become principal components.
Examples by features
Remove the mean
Measure shared variation
Find main directions
Represent data in fewer dimensions
2. Why variance matters
If a direction has high variance, points are spread out along it. PCA keeps high-variance directions because they preserve more structure under a linear projection.
Original space
Many dimensions, often hard to inspect.
PCA space
Fewer dimensions that preserve dominant variation.
3. AI/NLP example
Embedding spaces are high-dimensional. PCA can project word, sentence, or document embeddings into 2D or 3D for inspection. It will not fully explain a neural representation, but it can reveal clusters, domain shift, outliers, and artifacts.
4. Cautions
| Caution | Meaning |
|---|---|
| PCA is linear | It may miss nonlinear structure |
| High variance is not always semantic | Frequency or style can dominate |
| Visualization can mislead | 2D projection loses information |
| Scaling matters | Feature scale can change components |
Takeaway
PCA is useful because it turns variance into representation. For AI research, it is often an exploratory lens, not final proof.
References and learning path
This note uses the statistics-to-machine-learning route that fits my AI/NLP research goals: build intuition with Seeing Theory and StatQuest, connect it to Python practice with Think Stats, then deepen the ML connection with ISLR/ISLP, CS229, and selected statistical inference references.