· Xiaojing Yang · NLP and LLMs · 2 min read
中文Word Embeddings and Sentence Embeddings
How discrete language becomes vector space, and why sentence embeddings matter for retrieval and evaluation.
Core idea
Embeddings turn symbolic text into geometric objects that models can compare, transform, and retrieve.
1. From words to vectors
Words are discrete symbols. Neural models need numbers. Embeddings map tokens, words, sentences, or documents into dense vectors where similarity can be computed.
one model token
lexical meaning
sentence-level semantics
longer context
nearest neighbors
2. Why sentence embeddings are different
A word embedding represents a lexical item. A sentence embedding tries to compress context, syntax, topic, and meaning into one vector. That compression is useful but lossy.
| Use case | Embedding level |
|---|---|
| Similar words | word embeddings |
| Semantic search | sentence/document embeddings |
| RAG retrieval | passage embeddings |
| MT evaluation | multilingual sentence representations |
3. Hugging Face practice
from transformers import AutoTokenizer, AutoModel
name = "sentence-transformers/all-MiniLM-L6-v2"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModel.from_pretrained(name)4. My research connection
Embeddings connect directly to RAG, COMET-style MT evaluation, multilingual similarity, and error analysis. In petroleum-domain MT, sentence representations can reveal whether technical meaning is preserved even when surface wording changes.
Similarity view
Vectors close together are treated as semantically related.
Caution
Embedding similarity can miss factual grounding, domain terminology, and negation.
Takeaway
Embeddings are powerful because they make language measurable, but the measurement is never the whole meaning.
Interview pattern
My interview answer would usually be:
- define the concept in one sentence;
- explain the data flow;
- name the main failure mode;
- connect it to evaluation, multilinguality, or fine-tuning.
References
- Hugging Face Course
- Hugging Face Transformers documentation
- Hugging Face tokenizer summary
- Hugging Face fine-tuning guide
- Hugging Face PEFT
- The Illustrated Transformer
- Speech and Language Processing, Jurafsky & Martin
- Stanford CS224N readings
- Attention Is All You Need
- COMET: A Neural Framework for MT Evaluation