NLP system development: where accuracy drops
A ready-made model from the open Hugging Face repository shows classification quality above 0.9 on a demonstration set — and this is usually where the pilot stops.
The validation set is assembled from short, clean texts, matches the training distribution, and the labels are assigned by one clear rule. In real traffic, the same classes yield noticeably lower model accuracy, and the gap is not explained by the model.
Processing incoming requests brings mixed input: typos and transliteration, chat fragments, emails with conversation history and signatures, two or three languages within a single message.
On top of that comes internal terminology — product abbreviations, tariff names, department jargon — that is absent from any open corpus, so the model confidently assigns such texts to the wrong class.
The third cause is data labeling. If the annotation instructions describe classes with examples rather than decision rules, agreement drops: on borderline cases labels diverge, and the model learns to reproduce these disagreements.
Then drift kicks in — a new product appears or the wording of requests changes, accuracy declines, and without a separate control loop this is noticed only from complaints.
That is why NLP system development starts not with model selection but with a data profile: we measure text length, the share of mixed languages, typo frequency, and compile a dictionary of the client's terms.
Then comes a labeling scheme with rules and arbitration for disputed cases, a holdout set for monitoring, and regular checks on live traffic. We explain how this is addressed at each step.
What a well-built text processing pipeline provides
Requests resolved without a human are not "full automation" but the share of queries falling within the model's confident interval. We calculate it on a labeled sample and tune the threshold so that disputed cases go to an operator, not to the client. Throughput grows without expanding staff: the same people handle the peak queue.
Response time consists of model operation and post-processing. Latency is more often caused not by it but by synchronous calls to reference data and writes to the record system.
By measuring each stage, we remove unnecessary waits: the first meaningful response goes out in fractions of a second, while heavy checks run in the background.
Downtime is expensive when text parsing is embedded in a single point.
Therefore, models and validation rules are versioned, processing runs on several instances, and in case of failure the pipeline switches to a simplified mode — rule-based sorting and handoff to an operator instead of service failure.
Quality control is regular checking on a holdout sample, not a report after launch. We calculate the error rate by class: entity extraction, intent detection, sentiment — and make sure growth in one class is not masked by the system-wide average.
Separately, we track the failure class that has been eliminated: for example, emails that were lost because of format or language.
Ready-made models and libraries from the ecosystem such as Hugging Face cover the base layer of the NLP pipeline, so time goes not into training from scratch but into labeling for your domain and error analysis. This is what determines what share of requests are actually resolved without a human.
Implementation options: cloud, in-house infrastructure, hybrid
The choice of architecture is dictated by three things: data sensitivity, the volume of labeled examples, and the rate of change in the domain. Below are four alternatives that teams actually choose between at project start.
| Option | When it fits | Limitations |
|---|---|---|
| Ready-made model | Quick start, typical tasks, little own data | Quality hits the ceiling of someone else's domain: terms and internal designations are recognized inconsistently |
| Model fine-tuning | There are several thousand agreed examples and a clear quality metric | Regular labeling is needed; when data changes, quality drops and a new cycle is required |
| In-house infrastructure loop | Data cannot leave the perimeter, full control over logs is required | Longer launch: redundancy, monitoring, and model updates inside a closed network |
| Hybrid architecture | Some requests are insensitive, some are strictly internal | Harder to maintain: two loops, format alignment, request routing |
Data requirements are the same for all options: a unified format, consistent labeling, and a fixed rule for what counts as a correct answer.
Usually, teams start with a ready-made model — it provides a working prototype on real requests, and it immediately shows where quality is lacking.
Fine-tuning is added when examples for your own classes have been accumulated, while an in-house infrastructure loop or hybrid is chosen when the perimeter matters more than launch speed.
The architecture is not chosen once and for all: as request volume grows and labeled data accumulates, moving between options is planned work, not rewriting the system from scratch.
Implementation stages: from data audit to deployment
A language processing project more often fails not on the model but on a data mismatch: in development, a clean export; in production, live traffic with typos, mixed languages, and empty fields. That is why we follow a fixed sequence where each step has a verifiable result.
- Data audit. We review sources: volume, share of unlabeled records, duplicates, personal data. The output is the suitability of each source and a list of what we clean manually.
- Solution architecture. We determine where the model fits into the loop: what is handled by rules and dictionaries, what by training, and what input and output format neighboring systems need. We agree on the architecture before start so integration does not have to be redone.
- Reference set. We assemble a manually labeled set covering rare and disputed cases, and record metrics for each class rather than one overall figure. This is the comparison baseline for all subsequent versions.
- Setup and training. We prepare features and labels, train the model, and tune trigger thresholds to the acceptable false rate. Each version is saved together with its parameters and its result on the reference set.
- Live traffic validation. We run in shadow mode: the model responds, while the current process makes the decision. We compare with the reference, look at the share of low-confidence answers, latency, and behavior on rare classes.
- Deployment. We move to the production loop with model versioning, request logging, and rollback to the previous version. We set up drift monitoring: quality degradation is visible before the client notices it.
- Handover. Documentation, retraining and error analysis procedures, access — the client's team develops the solution further without us.
Deliverables: configurations, diagrams, procedures
The deliverables are what makes it possible to operate and develop the system without our involvement: environment configurations, data processing diagrams, labeling rules, documentation, and procedures.
Everything listed is transferred to the client's repository and accepted together with the code — not as a separate "document package" that is opened once at handover and never again.
- Environment configurations: deployment, environment variables, resource limits, access to external services — with an explanation of what each parameter affects and what happens if it is changed.
- Processing diagram: the request path from intake to response — where the model is, where filter rules are, where the cache is, which points are logged, and how to link a log entry to a specific system decision.
- Labeling rules: instructions for annotators, a dictionary of disputed cases, criteria for agreement between annotators, and a procedure for resolving conflicts in assessments.
- Model documentation: feature composition, provenance of the training set, quality metrics by class, known applicability boundaries, and typical errors on live data.
- Operations procedures: what we monitor, quality degradation thresholds, the procedure for rolling back to the previous version, dictionary update frequency, and retraining rules.
- Alerting scheme: which metrics are collected, where to view them, at what values the on-call engineer receives a signal, and what they do as the first step.
- Solution handover procedure: access and roles, starting the system from scratch in a clean environment, and instructions for the client's engineer who joins the project after us.
Case study: request processing in a support service
Customer support received several thousand requests per month, and all of them went to manual processing. The operator determined the topic and urgency, searched the knowledge base for an answer, and decided which line to route the request to.
At peak, the client received the first meaningful message after several hours, and about a fifth of requests were forwarded between lines — each transfer added delay and a repeated explanation of the problem.
We assembled a pipeline: request text processing, topic and urgency detection, search for a suitable knowledge base fragment, automatic response when model confidence is above the threshold, and routing to the right line when confidence is below.
The operator's workspace was connected to Zendesk: the employee sees the proposed topic and response draft and confirms or edits them in one action instead of composing the answer from scratch.
| Metric | Before | After |
|---|---|---|
| Share of automatic responses | 0% | 63% |
| Time to first response | hours | minutes |
| Incorrect routings | 18% | 4% |
| Requests closed manually | 100% | 37% |
The confidence threshold is the main setting in such a system: a lower threshold means more automatic responses and more errors, so we tuned it on a labeled sample of real requests, not on the holdout test set.
The model does not "guess" disputed cases; it hands them to an operator together with the fragment it relied on — this eliminates the failure class where the client receives a confident but incorrect answer.
How does fine-tuning an NLP model differ from training from scratch?
The difference is the starting point. Training from scratch initializes weights with random values, and the model derives language patterns only from the corpus it was shown.
Fine-tuning starts from a pretrained model, where syntax, morphology, and general semantic relationships are already captured, and adapts it to a specific task — request classification, entity extraction, dialogue parsing.
Hence the difference in sample size by orders of magnitude. For domain fine-tuning, a few hundred labeled examples per class are usually enough for metrics on the holdout set to reach a working plateau.
Training from scratch requires a corpus of billions of words and tokens and distributed computation over weeks — and on a narrow domain it still often loses to model transfer.
Training from scratch is justified when the domain radically differs from the training corpus: a rare language without digital texts, narrow notation such as equipment telemetry or genetic sequences, or the client's requirements prohibit the use of external pretrained weights for licensing reasons or data privacy requirements.
In practice, a hybrid more often works: first, continued pretraining on your own unlabeled corpus — it pulls the vocabulary and distributions closer to the domain lexicon — then fine-tuning on a small labeled sample.
Such model transfer reduces the need for labeling by about two orders of magnitude, and we validate quality on a holdout set of real domain data.
That is why we first assess whether fine-tuning is enough to reach the required quality, and only then discuss the corpus and labeling for training from scratch.
Technical concerns: data, labeling, load
Engineering concerns are not about the model but about everything around it: whether there is enough labeling, whether the solution can withstand peak load, whether data will leak, and who is responsible for quality in six months. Below are answers to what most often comes up during technical acceptance.
How much labeling is needed for the model to work?
We calculate the volume based on the task: at the start, a few hundred labeled examples per class are enough; later we add more on disputed cases from live traffic.
More important is the quality of the scheme — double labeling part of the sample and measuring agreement shows whether the instructions are contradictory. If annotators diverge, fine-tuning will not help: first we fix the rules.
What happens under peak load?
We capture the traffic profile before design: peak hours, seasonal spikes, share of heavy requests. For peaks, we build in a queue and priorities so that the system degrades — responds more slowly — instead of failing. We run load tests against the real profile, not the average.
How is data privacy ensured?
We deploy the model in the client's environment: data does not leave the infrastructure and is not used to train third-party solutions. We anonymize personal fields before they enter the sample, access to raw data is restricted, and calls to the model are logged.
Who is responsible for quality drift after launch?
We catch drift with live traffic metrics, not complaints: we track the share of confident answers and refusals and collect disputed examples. We trigger fine-tuning when a metric crosses the threshold, and afterward run the regression set so as not to break what was working. The update procedure and on-call handover are included in the deliverables.
Let's discuss the task: what to send for an estimate
To come back with a project assessment and work plan rather than generalities, we need a minimum of context. You will not have to prepare documents: what the team already has is usually enough — request exports, a process draft, sample emails. The closer these materials are to reality, the fewer revisions will surface during acceptance.
- Data examples — several hundred real requests, emails, or transcripts in the form in which they enter the system: with typos, mixed languages, attachments, and empty fields.
- Process description — where the text comes from, who does what with it next, and which systems receive the result: ticketing, internal portal, reporting.
- Quality requirements — which errors are unacceptable and which are tolerable, whether confidence estimation is needed, and what the system should do when it is low.
- Perimeter constraints — where data is allowed to be processed: external cloud, secure environment, or fully local deployment, and how long it can be stored.
- Volumes and mode — how many documents arrive per day, whether a real-time response is needed or scheduled processing is sufficient.
- Acceptance criteria — on which sample of examples we check quality and what deviation from expectations we consider acceptable for launch.
This is enough to assemble a work plan: stages, team composition, timelines, and points where you see an interim result. The conversation takes about an hour — as a result, you receive an assessment with explicitly stated assumptions and open questions, not a presentation.
After that, we agree on the start of work and the cadence of demonstrations so that the result is verifiable at every step.
