LoRA for Neural Machine Translation

中文导读

LoRA 经常被一句话概括成“参数高效微调”。但如果要真正用好它，尤其是在 neural machine translation 里，我觉得需要理解三个问题：

1. LoRA 到底改变了模型里的什么？
2. 为什么它适合 domain adaptation？
3. 它和 full fine-tuning 到底在 trade-off 什么？

这篇文章用 English—Norwegian petroleum MT 的背景来解释 LoRA，而不是把它当成一个抽象技巧。

The basic idea

Full fine-tuning updates all model parameters. For a large translation model, this can be expensive, unstable, and hard to repeat many times.

LoRA takes a different approach. Instead of updating the full weight matrix, it freezes the pretrained model and learns a small low-rank update.

Conceptually, if a model has a weight matrix $W$, LoRA keeps $W$ frozen and adds a learned update:

Instead of learning a full $\Delta W$, LoRA decomposes it into two smaller matrices:

where the rank $r$ controls the size of the update space.

In practice, the update is usually scaled:

This is where the LoRA scaling factor $\alpha$ becomes important.

Why low-rank adaptation can work

The intuition is that domain adaptation may not require changing the entire model. A pretrained MT model already knows a lot about language, translation, and syntax. For a specialized domain, we often need to steer it toward domain terminology, preferred phrase mappings, formal style, recurring sentence structures, and target-language conventions.

These changes may live in a much smaller subspace than the full model parameter space. LoRA is useful because it lets us learn that adaptation signal without rewriting the whole model.

Why LoRA fits low-resource domain MT

Low-resource domain MT has two constraints:

1. limited high-quality parallel data;
2. limited compute for repeated experiments.

Full fine-tuning can adapt strongly, but it updates all parameters. With small data, that can increase overfitting risk and make experiments expensive. LoRA updates only a small fraction of parameters, which gives three practical benefits:

• lower memory cost;
• easier repeated tuning;
• more stable adaptation under limited data.

In my petroleum MT project, this mattered because the cleaned dataset had fewer than 20,000 sentence pairs. The goal was not to train a new translation model from scratch, but to adapt an existing multilingual model to a domain.

Where LoRA is inserted

In many NMT setups, LoRA is applied to attention projection layers such as query, key, value, and output projection.

Attention layers are central to token interactions, which makes them a reasonable target for translation adaptation.

One could also apply LoRA to feed-forward layers, but that increases the number of trainable parameters. In a low-resource setting, more trainable capacity is not automatically better. It may increase cost and overfitting risk.

Important hyperparameters

The most visible LoRA hyperparameters are:

• rank $r$: the dimensionality of the low-rank update;
• alpha $\alpha$: the scaling factor for the update;
• dropout: regularization applied inside the adapter;
• target modules: which layers receive LoRA updates.

In my experiments, $r=8$, $\alpha=64$, and dropout $=0$ emerged as a strong configuration. The interesting part was not only the final setting, but the fact that alpha dominated the hyperparameter landscape.

This suggests that, in this domain adaptation setting, the magnitude of the update mattered more than simply increasing the rank.

LoRA vs full fine-tuning

Full fine-tuning gives the model maximum flexibility. LoRA gives the model controlled flexibility.

That distinction matters.

With very small data, full fine-tuning may not have enough reliable signal to use its full capacity. With moderate data, it may gain a small advantage. But if the task saturates quickly because the domain text is regular and terminology-driven, LoRA can approach full fine-tuning performance with far fewer trainable parameters.

This was exactly the pattern I observed: LoRA was competitive across data scales while being much more parameter-efficient.

What LoRA does not solve

LoRA does not automatically fix noisy data, wrong alignments, missing terminology, bad evaluation design, train/test leakage, or domain mismatch between corpus and deployment setting.

This is why LoRA should not be treated as a magic adapter. It works best as part of a pipeline that includes data diagnostics, careful splitting, tuning, and multi-metric evaluation.

Takeaway

LoRA is useful for neural machine translation because it gives us a practical middle ground:

• more adaptable than zero-shot translation;
• cheaper and more stable than full fine-tuning;
• flexible enough for domain terminology and style;
• controlled enough for low-resource experiments.

For domain MT, I see LoRA less as a shortcut and more as an experimental tool: it lets us study how much adaptation is possible when data and compute are limited.