· Xiaojing Yang · NLP and LLMs · 2 min read
中文Transformers for NLP
How Transformers combine self-attention, feed-forward layers, residuals, and positional information.
Core idea
A Transformer is a stack of attention-based representation updates.
1. The architecture idea
Transformers replaced recurrence with self-attention and parallel computation. Each layer updates token representations using information from the sequence.
input vectors
mix contextual information
stabilize updates
transform each position
repeat
2. Encoder, decoder, encoder-decoder
| Family | Typical use |
|---|---|
| Encoder-only | classification, NER, sentence representation |
| Decoder-only | language modeling, chat, generation |
| Encoder-decoder | translation, summarization, text-to-text tasks |
3. Hugging Face practice
from transformers import AutoTokenizer, AutoModelForSequenceClassification
name = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name, num_labels=2)4. My research connection
Transformers are the common backbone behind multilingual encoders, MT systems, COMET-like metrics, and LoRA fine-tuning. Understanding the block helps explain where adapters can be inserted and why tokenization affects everything downstream.
Engineering view
Use pretrained checkpoints and task heads.
Research view
Ask what representation, language coverage, and adaptation mechanism the architecture supports.
Takeaway
Transformers are not one model. They are a reusable architecture pattern for contextual representation and generation.
Interview pattern
My interview answer would usually be:
- define the concept in one sentence;
- explain the data flow;
- name the main failure mode;
- connect it to evaluation, multilinguality, or fine-tuning.
References
- Hugging Face Course
- Hugging Face Transformers documentation
- Hugging Face tokenizer summary
- Hugging Face fine-tuning guide
- Hugging Face PEFT
- The Illustrated Transformer
- Speech and Language Processing, Jurafsky & Martin
- Stanford CS224N readings
- Attention Is All You Need
- COMET: A Neural Framework for MT Evaluation