Dataset augmentation methods in LLM training

Why data augmentation breaks on real datasets

Data augmentation usually breaks not at the stage of example generation, but earlier — during corpus analysis.

If texts are collected from a single channel, for example only from support conversations, the dataset develops sampling bias: the length of utterances, the share of formal requests, and the range of topics change.

Rare phrasings and edge cases make up fractions of a percent, and during splitting they almost entirely end up in the validation part.

Then model overfitting kicks in, which is not visible in the metrics report.

When there is little training data, the model most quickly learns a stable phrasing pattern rather than the meaning of the request: substitute another topic into the same phrase — the answer will not change.

The validation part does not show this because it is collected from the same distribution as the training part.

The third typical failure is noise in the training set. Automatic paraphrases on short utterances change the meaning, duplicating rare classes gives the model a surrogate feature instead of a real one, and shuffling fragments breaks the coherence of the dialogue.

On live requests, quality diverges from validation, and the divergence is greater the more aggressive the generation was.

Next, we discuss how this is addressed: by sampling stratification, selecting examples by distribution divergence, and controlling noise before feeding into training.

What changes after implementation: effect in measurable metrics

The effect of controlled augmentation is visible on a holdout set of real requests, where some questions are deliberately paraphrased.

We calculate model quality metrics by semantic groups, not as a single overall number: this shows where the model understands the request and where it guesses a familiar phrasing.

We test robustness to paraphrases directly — we ask the same question in several formulations and look at the variance of answers.

Easy Data Augmentation — synonym replacement, swapping, and word deletion — gives the cheapest gain on this measurement: the model stops depending on word order and on one successful phrase in the training set.

The volume of manual labeling decreases not because we label less data, but because we label different data. Typical cases are covered by augmentation, so people spend time on contentious examples and consistency control. The share of tasks requiring a human decreases, and agreement between annotators increases.

We measure training stability as the variance in quality between runs on the same dataset. Balanced augmentation removes skews in rare classes that made the model jump from run to run. Generation rules and dataset composition are fixed — a repeated run reproduces the same picture.

Result: answers stop depending on the wording of the question, the share of manual labeling decreases, and model updates stop being a lottery. The solution can be improved iteratively without rebuilding the training set from scratch.

Comparison of approaches: rules, models, and synthetic data

Corpus expansion methods differ not in complexity but in the cost of error: some add variability in phrasing but not new meanings, others generate new examples — along with new noise.

Below is a comparison by applicability and limitations, to choose a method for the volume and type of data.

What to consider when choosing

In practice, methods are combined, but no more than two or three per corpus: rules provide volume at the start, generative synthesis provides diversity of phrasing.

Before adding to training, the dataset is deduplicated and checked for overlap with the holdout part, otherwise quality evaluation stops showing anything.

The share of synthetic data is kept within one third — beyond that, the model begins learning from its own formulations, and quality on real data drops.

How data augmentation is built: stages of engineering work

Data augmentation is not “multiplying examples,” but controlled corpus expansion with quality control and no overlap between datasets. The work proceeds in stages: each ends with an artifact and a verifiable criterion, not a report that “we made more data.”

1. Corpus audit. Inventory of sources: volume, languages, domains, share of duplicates and corrupted examples, labeling state. The output is a report and a list of scarce example classes that actually need to be synthesized.
2. Preparation pipeline schema. We fix the order of steps: normalization, deduplication, expansion, filtering, splitting into training, validation, and test parts. The schema is agreed upon before launch: edits at this stage cost hours, after — retraining cycles.
3. Setting up rules and generation. Deterministic transformations (templates, permutations, back translation) plus synthetic generation by a model. We set constraints on length, language, and style and a limit on the synthetic share — usually 20–30%, otherwise the model reproduces the generator’s artifacts.
4. Quality acceptance criteria. We fix thresholds in advance: share of examples rejected by the filter, share of duplicates, labeling consistency, preservation of class distribution.
5. Leakage check. We look for overlaps between training and validation parts by text fragments and embedding proximity, and cut out what is found. Without this step, the validation metric is inflated, and quality degrades in production.
6. Effect measurement. Control training on the original and expanded corpus with identical validation sets. We look not only at the overall metric but also at subsets where data was initially scarce.
7. Handover. Configurations, reproducibility scripts, pipeline schema, and a report with metrics; versioning of the corpus and rules so the next run repeats without manual assembly.

Handover artifacts: pipeline schemas, quality rules, regulations

After the work is delivered, the client should retain not a presentation report, but a working kit that the team can use to continue without us.

All project artifacts sit next to the code, are versioned, and go through review just like source files — otherwise, a month later no one will remember why the filtering threshold was set exactly that way.

• Pipeline schemas — a graph of augmentation stages as configuration (YAML/JSON): step dependencies, branching points, input and output formats at each stage.
• Data documentation — dataset passport: sources and licenses, field schema, distribution of classes and duplicates, share of synthetic examples relative to original ones.
• Labeling rules — instructions with edge cases, worked examples, and escalation procedure for contentious fragments; a new annotator can get up to speed without oral explanations.
• Quality control regulation — agreement metrics between annotators, share of examples for re-check, and the threshold below which a batch is sent for rework.
• Filtering configurations — deduplication rules, thresholds for vector representation proximity, blacklists; parameters are externalized, and edits do not require code changes.
• Reproducibility scripts — fixed environments and generator seeds: a repeated run yields the same dataset, and any discrepancy is explained, not hunted for.
• Measurement report — comparison of the model before and after augmentation on the holdout set, broken down by subgroups and with an honest indication of where the effect was not confirmed.
• Decision log — which approaches were tried, which were rejected, and why.

Case study: model robustness on rare phrasings and noise

The client is a service with a first-line assistant: on typical questions, quality was satisfactory, but as soon as a user rephrased a question, added slang, or made a typo, the share of correct answers dropped.

On the holdout set of rare phrasings, it fluctuated in the 61-72% range from run to run, and about one third of answers had to be corrected manually — support worked as a second contour of the model.

Rare phrasings were extracted from request logs and labeled as a separate layer, without mixing them with the typical stream.

Augmentation proceeded in several passes: back translation, meaning-preserving paraphrases, entity and number substitution, and generation of examples for rare templates.

Each example passed a filter — label match, absence of duplicates, lexical diversity; rare classes were balanced so that data augmentation did not dilute frequent ones.

Metrics before and after: share of correct answers on rare phrasings 61-72% → 88-91%, variance between training runs 11 → 3 points, manual corrections 32% → 8%.

On the overall evaluation, including RussianSuperGLUE, there was no degradation — robustness was not bought at the cost of other quality.

We fixed the protocol: the dataset is fixed together with filtering rules, measurement is performed on the holdout set across five runs, and changes in data are visible in comparison, not in gut feelings.

How fine-tuning on weak labeling differs from the full cycle

Fine-tuning a model on already collected labels is adapting pre-trained representations to your domain and answer format.

This path is appropriate when the task is close to the one the model was trained on, and labels within a class are consistent with each other: the model does not relearn the language but shifts the answer distribution. The work takes days, and all you need is a labeled corpus and a holdout set for control.

Weak labeling stops being a foundation when the rules themselves diverge.

If annotators draw the class boundary differently, and rare cases are dumped into a general “other,” fine-tuning faithfully reproduces the noise: the holdout metric rises, while quality on the real stream drops. Data augmentation does not help here — it multiplies the erroneous label across new phrasings.

The full training cycle begins with rebuilding the corpus: revising the class schema, annotator instructions, control examples, and measuring agreement between people.

At the input, weak supervision schemes are useful — Snorkel with a set of labeling functions provides draft labels and an estimate of their reliability, but the final layer still undergoes manual review.

Choosing a strategy is diagnostics, not taste. We calculate annotator agreement, analyze errors by class, and look at the gain on the control set: as long as fine-tuning yields a gain, the full cycle is not needed. When the model hits the ceiling set by label quality, only rebuilding the data moves it further.

In terms of labor costs, the fork looks like this: fine-tuning — days and a safe rollback to the previous version; the full cycle — weeks and a managed labeling scheme that is reused in subsequent iterations.

Technical doubts: noise, leaks, quality control

Augmentation is tested on a small sample, but questions about it arise later — when the expanded dataset goes into training. Below are four technical doubts that we resolve with checks, not promises.

How are noise and repetitions filtered out?

We catch exact duplicates by hashes, near-duplicates by embedding distance with a threshold tuned on a labeled sample. A model-based filter assesses coherence and answer-to-question relevance; borderline examples go to manual review. We track the duplicate share as a separate metric: its growth means the generator is degenerating.

What is included in dataset quality control?

On each run, we calculate the share filtered out, length distribution, overlap with the control part, and agreement between model labeling and manual labeling on a stratified subsample.

Thresholds are fixed in configuration, not “by eye”: a dataset that fails checks does not go into training.

Will augmentation distort the class distribution?

Balancing rare classes is worthwhile, but rewriting the real frequency is not: otherwise the model confidently answers things that almost never occur in production.

We maintain stratification by source, length, and request type and compare shares before and after expansion — a shift beyond tolerance blocks the dataset.

Let’s discuss the task: where to start work on your corpus

Data augmentation begins not with choosing methods, but with analyzing the corpus: what is already in storage determines whether paraphrasing, back translation, or synthetic generation is needed.

A corpus audit takes less time than subsequent debugging — it reveals where labeling is contradictory and where duplicates inflate the dataset and distort metrics.

To come back with an estimate and implementation plan, rather than clarifying questions, we need the inputs below. Discussion of the task is free-form: for each item, whatever you have at hand is enough.

• Goal and metric: what task the model solves and by what indicator you will see that quality has improved.
• Corpus volume and structure: how many examples, in what language, in what format texts and labels are stored.
• Origin of labels: how much data was labeled manually and according to what instructions — this determines whether the labels are suitable as a reference.
• Classes and imbalance: which categories are so rare that the model will not learn them without augmentation.
• Domain constraints: what cannot be changed in texts — names, terms, legal wording, numbers.
• Previous runs: what data was already used for training, what result was obtained, and how the run is reproduced.
• Access and acceptance: where the corpus is hosted, whether there are export restrictions, and who accepts the result.

Work begins after this analysis: we fix which augmentation methods we apply, on what part of the data we validate, and how we measure gain on the holdout set. As output, you receive a work plan and acceptance criteria agreed upon before the first training run.