Training a Text Classification Model (BERT, RoBERTa, DeBERTa)

Why BERT fine-tuning on your own data stalls

A rule-based model maintains predictability on template requests, but breaks on paraphrases and typos. BERT fine-tuning on your own data should close exactly this gap — and this is where classification quality most often drops. The reason is usually not the choice of architecture, but the material the model learns from.

Your own labeling almost always arrives raw. Different annotators read the instructions their own way, and the same complaint lands sometimes under “payment,” sometimes under “delivery”; duplicates get different labels, and long texts are labeled by one sentence out of five.

The model sees contradictory examples and learns to average them — on borderline phrasings, accuracy doesn't grow no matter how many epochs you add.

The second trap is imbalance. If a rare category makes up fractions of a percent of the corpus, it's more profitable for the model to always output the majority label: aggregate accuracy looks decent, but the target class isn't found at all, and losses hide in the average.

The third is validation on the same labeling as training: the number looks nice, but quality drops in the live stream because disputed and hard cases simply didn't make it into the set.

We'll break down how this is closed: first the class schema and dispute resolution rules, then inter-annotator agreement control on an overlap sample, balancing rare categories, and a separate holdout set collected from real traffic. Only after that does BERT fine-tuning make sense — otherwise the model will neatly learn noise.

What effect a model trained on your own data delivers

BERT fine-tuning on your correspondence changes not abstract “model quality” but shares in the request flow. We label several thousand real tickets, emails, and chat messages according to your taxonomy and immediately see which pairs of categories are confused most often.

Then fine-tuning on your own labeling: classes become yours, not “general” — a delivery complaint, access request, and payment failure stop merging into one pile.

We run it on the current branch — Hugging Face provides ready weights and tooling, so time goes into data and labeling, not building architecture from scratch.

The share of requests handled without a human usually rises from half to three quarters.

The remainder doesn't fall to an operator “as is” but goes into a queue with a ready category and priority: a human confirms or corrects, and every correction returns to the dataset for the next iteration.

We calculate model quality metrics by class, not one overall number: precision and recall for problematic categories, confusion matrix, confidence threshold below which a request goes to review. This shows what to fix next — data, labeling, or class boundaries.

The eliminated class of errors was confident “wrong-way” answers when the text is formally similar but the meaning is different. Labeling time also drops: the model itself highlights disputed examples at class boundaries, and the next iteration requires less manual work.

Approaches to BERT fine-tuning: comparison of modes

Before start, we fix three parameters: how many labeled examples per class actually exist, how much domain vocabulary diverges from general language, and what matters more — quality on rare classes or response time.

The mode depends on them, and almost always it isn't just one: part of the task is covered by the model, part by deterministic rules.

The boundary between modes is checked on the same holdout set, not by eye.

Fine-tuning usually wins with dozens of classes and limited labeling; training from scratch is justified only for a truly distant domain, and a hybrid gives the largest gain where part of the signal is formalizable.

Implementation stages: from labeling audit to handover

The order of stages is not arbitrary: each next one relies on the artifact of the previous. If you skip the labeling audit and immediately launch BERT fine-tuning, the model will learn the annotators' mistakes, and tuning won't fix it anymore. Below is the path the work follows.

1. Corpus and labeling audit — share of contradictory examples, duplicates, class imbalance, annotator agreement, whether texts belong to the target domain.
2. Metric definition — what is more critical: recall on a rare class or precision on a mass class. We fix a baseline on a simple model so the neural network has an honest benchmark.
3. Class schema — hierarchy of categories, rules for disputed cases, minimum examples per class. We agree on it with those who will review predictions manually.
4. Fine-tuning — choice of pretrained model and window length for typical text, class balancing, early stopping on the validation set, not on the number of passes.
5. Model validation — holdout stratified sample, error analysis by class, and a separate run on texts from another period: this reveals data shift that the average metric hides.
6. Deployment into the process — service with model versioning, prediction logging, and a confidence threshold: disputed cases go to a human, not to automation.
7. Handover — fine-tuning code, configs, rerun checklist, class schema documentation, and operator instructions for reviewing questionable outputs.

Project artifacts: model weights, labeling guidelines

The client is left not with a “model in an archive” but with a kit that the team can use to repeat training, check quality, and continue labeling on their own.

We hand over four groups of artifacts: weights and launch configuration, model documentation, labeling guidelines, and operator instructions.

• Weights of the fine-tuned model and inference configuration: confidence threshold, label-to-class mapping, base architecture branch — deploys on a new server without contacting us.
• Model documentation: class schema, size and sources of the training sample, selected training parameters, fixed limitations by domain and text type.
• Quality report on the holdout set: precision and recall per class, confusion matrix, list of typical false positives with examples.
• Labeling guidelines: definition of each class, review of borderline cases, rule for resolving disagreements between annotators.
• Operator instructions: step-by-step labeling procedure, a short set with reference answers for training newcomers, self-check checklist.
• Data preparation and validation scripts: duplicate detection, class imbalance assessment, filtering out corrupted and too-short texts.
• Retraining procedure: from raw data to a new model version, with the same metrics for comparing versions.
• Acceptance protocol: fixed quality thresholds, rollback procedure to the previous version, persons responsible for acceptance.

Such handover removes the main risk — dependence on the contractor. The customer sees what data the model was trained on, where the boundary of its applicability lies, and by what procedure to train a new class when the request flow changes.

Support request classification: metrics before and after

Support request flow went through a single queue: operators manually routed them by topic, and urgent outage messages stood in the general mass. The share of manual sorting stayed around 100%, and time to first response depended on who would process the queue first.

We started with labeling: collected request history over several periods, reduced topics to a stable class vocabulary and removed intersections that made one request fit two categories.

The baseline model on BERT gave macro-F1 0.61 on the holdout set — for routing without a human this is not enough, so fine-tuning ran on balanced classes with separate quality control on rare topics.

What changed during the project

Key improvements came not from training settings but from data: eliminating contradictions in labeling, adding examples for rare topics, and a confidence threshold below which a request goes to an operator.

False positives became predictable — the model flags questionable cases rather than staying silent about them.

What metrics showed after implementation

macro-F1 on the holdout set rose from 0.61 to 0.88, the share of automatically routed requests — up to 74% of the flow.

Time to first response in urgent categories decreased roughly threefold, and manual sorting was reduced to isolated cases: customer support stopped being a bottleneck at intake.

What to do if a rule- and dictionary-based model is already in place

A full migration is not needed and almost always harmful. The working scheme is a hybrid approach: dictionaries and rules continue to cover simple cases, and the trainable model connects to what they don't cover.

Both branches live on the same input, and the decision about which one gets the text is made by a confidence threshold.

The route is built as follows: first the dictionary runs — an exact match by template or stop list closes the question without the model. Everything the rules didn't parse or parsed with low confidence goes to the model. Every divergence is logged — this is ready material for the next training iteration.

Separately preparing a corpus is usually not required: the history of rule runs already contains class labels. The first version is trained on it, and disputed cases where the rule and model diverge go to manual review and return to the sample.

This way quality grows where it is critical, not across the entire dictionary at once. Fine-tuning BERT and related architectures here is not rewriting the system but replacing one of the decision blocks.

Migration without stopping the process is done with shadow mode. The new model scores live traffic for a while, but its answers are only compared with the rules and don't affect output.

When divergence from the reference fits within a given threshold, we open a share of traffic — first a small one, then all of it. Rollback is flipping a flag; dictionaries stay in place and continue working.

As a result, rules become insurance for rare classes and rigid requirements, while the model carries the main load. The share of manual reviews drops severalfold, dictionaries shrink to truly critical cases — without stopping and without losing the quality already achieved.

Technical concerns: labeling, data volume, quality

The volume of labeling is determined not by a “magic” number but by the number of classes, their balance, and the diversity of phrasings.

For two or three stable categories, several hundred examples per each are usually enough; when there are dozens of classes and they overlap in meaning, the count runs into thousands, and part of the corpus is cheaper to collect with weak labeling plus manual review of disputed cases.

The gap between quality on the holdout set and in the real stream is more often a consequence of data divergence rather than a training error.

Therefore, we fix labeling rules before the start, clean out duplicates and nearly identical examples, and build quality control on error analysis: which pairs of classes the model confuses and in which phrasings this happens.

What to do if annotators contradict each other?

We don't smooth over contradictions but put them into a separate set: fix a rule for the disputed case and relabel borderline examples.

If two classes are truly indistinguishable by text, they are merged — otherwise the model learns to guess noise, and quality in production drops.

How to tell that the model has overfitted?

The sign is that the metric on the training set rises while on the holdout set it stops improving. Then we stop training based on the holdout set, reduce model complexity, and check with cross-validation across different data slices.

Separately, we look at robustness on short and atypical texts — overfitting shows up there earliest.

Let's discuss the task: what data you have and what is needed as output

For the estimate to be substantive rather than “by eye,” the first conversation needs three things: examples of your texts with or without labels, a list of target classes, and an understanding of how the result will be used — one-time processing of an accumulated archive or a continuous document stream.

This is enough to estimate the scope, choose a base model for BERT-family fine-tuning, and give timelines without guessing.

What we come back with after the review:

• assessment of the task: how many labeled examples are missing for stable quality and which classes the model confuses most often;
• work plan by stages with checkpoints — from corpus preparation to model handover;
• requirements for the customer's data: format, volume, labeling schema, anonymization rules;
• operating mode: where the model is hosted, how the class dictionary is updated, what happens when a new class appears;
• acceptance criteria in numbers: share of correct answers and confusion matrix per class on the holdout set;
• team composition and division of responsibilities during the project.

Project start does not require that the data already be prepared: we include corpus review and labeling in the plan rather than leaving them outside it.

Describe the task — what texts you classify, how many classes, and where the result should live — and we will come back with an estimate and work plan.