Data Scaling for LoRA Domain Adaptation

Data scaling is one of the most useful experiments in low-resource domain MT. It asks a simple but powerful question:

If I give the model more domain data, how does performance change?

This question matters because low-resource research often focuses on model choice, but the practical bottleneck may be data quantity, data quality, or domain coverage.

Why data scaling is research, not just bookkeeping

A data-scaling curve can reveal whether a method is truly useful in a low-resource setting.

If performance improves quickly with the first few thousand examples and then plateaus, the project may benefit more from data cleaning and evaluation than from collecting more similar data. If performance continues to grow, more domain data may still be valuable. If performance is unstable, the model may be sensitive to sampling, preprocessing, or hyperparameters.

For LoRA, data scaling also tests whether a small adapter has enough capacity to absorb domain information at different data sizes.

A practical experimental design

A clean data-scaling experiment might use subsets such as:

Each condition should keep the same:

• base model;
• preprocessing pipeline;
• validation and test sets;
• evaluation metrics;
• training budget rule;
• random seed policy.

Otherwise, the curve may reflect uncontrolled experimental changes rather than data scale.

What to measure

I would not measure only one score. A useful data-scaling table should include:

• BLEU;
• chrF;
• COMET;
• terminology accuracy or recall;
• validation loss;
• training time;
• seed variation;
• human error categories for representative outputs.

The goal is not just to find the highest point. The goal is to understand what kind of behaviour improves with data.

Possible patterns

1. Fast early gain

The first few thousand examples may teach the model repeated domain structures and frequent terminology. This is common when the domain has formulaic documents.

2. Diminishing returns

After a certain point, additional similar examples may produce smaller gains. This suggests that the model has already learned the dominant patterns, and remaining failures may require better data diversity or targeted terminology examples.

3. Noisy or unstable gains

If small subsets produce inconsistent results, the issue may be sample composition. One subset might contain many important terms; another might contain mostly repeated templates.

4. Metric-specific gains

BLEU may improve while terminology recall remains weak, or terminology may improve while COMET changes little. This is why multiple utilities matter.

Connection to LoRA

Data scaling is especially interesting for LoRA because the adapter capacity is controlled. A low rank may be enough for small data but saturate later. A higher rank may help with more diverse domain data but overfit in the smallest condition.

This connects data scaling to hyperparameter optimization:

• Does the best rank change with data size?
• Does alpha dominate across all scales?
• Does dropout help only in small-data conditions?
• Can one LoRA configuration transfer across data scales?

These questions turn HPO from tuning into research evidence.

Connection to my project

In English—Norwegian petroleum MT, data scaling can answer whether the cleaned corpus contains enough useful domain signal. If a small subset captures much of the gain, that suggests strong template regularity. If larger subsets improve terminology or reduce critical errors, then additional domain coverage matters.

This also helps interpret the value of data cleaning. A smaller clean subset may outperform a larger noisy subset, which would support the idea that low-resource MT is data-quality constrained, not only data-quantity constrained.

Limitations

Data-scaling experiments are easy to misread.

• Random subsets may not represent the full domain.
• Larger subsets may include more noise as well as more signal.
• Test sets may be too similar to the training distribution.
• Small metric gains may not be practically meaningful.
• One seed may hide instability.

So I would treat scaling curves as evidence, not as final proof.

Interview answer

I would use data-scaling experiments to study how LoRA adaptation behaves as domain data increases. The goal is to see whether performance improves steadily, saturates early, or depends strongly on subset composition. For my MT project, this is important because it separates three possible explanations: the model needs more data, the data needs better cleaning, or the evaluation metric is not capturing the domain behaviour I care about.