Why an NER Model Makes Mistakes on Real Data
In a demo, an NER model confidently finds companies and people — it seems the task is solved.
In a real stream of emails, chats, scans, and exports from accounting systems, entity recognition degrades, and it does so imperceptibly: there is nothing to compare the result against, and you cannot catch errors in a thousand lines by eye.
The first reason is a vague label schema. When “organization,” “department,” and “product” are lumped into one class, and nested entities are described by a single label, the model sees contradictory examples and guesses a compromise: entity boundaries drift.
The schema is rebuilt before training starts — flat labels, explicit nesting rules, a separate class for each entity that is actually needed in reporting.
The second is the quality of text annotation. If two annotators place boundaries differently, and disputed cases are resolved “by feel,” the model learns from noise.
Annotation in Label Studio with a double pass and discrepancy review gives a measurable agreement percentage — a corpus below the threshold does not go into training.
The third is rare entity classes. When the corpus is skewed, a rare label does not accumulate enough examples, and the model most often errs on exactly that label.
Format changes also hit separately: a new source brings different casing, scan recognition noise, tables — the distribution of input texts drifts away from the training distribution.
And the main thing is the lack of validation on a holdout set. Metrics on the training set say nothing about behavior on fresh data.
Validation on a holdout set of new arrivals that does not overlap with training gives an honest assessment and allows versions to be compared with each other — without it, any “improvement” is unverifiable.
We break down how this is addressed: from the label schema and annotation guidelines to the procedure for repeated validation on new data.
What Entity Extraction Delivers: Measurable Effects
Entity extraction pays off not through abstract automation but through three concrete shifts in the document processing workflow.
Share of Manual Review
The model labels entities by type — organizations, people, dates, amounts, details, contract numbers. The operator reviews only the fields where the model's confidence is below the threshold set for your process, rather than reading the document in full. The share of manual review drops from full reading to selective reading.
We tune the threshold using the error matrix: where a false positive is more critical than a miss, it is higher. This way, reviewers' attention goes to disputed cases instead of being spread evenly across the stream.
Document Processing Time
A document is received, labeled, and sent to the accounting system without manual field transfer. The first run over the archive is batch, then incremental as files arrive, so the queue does not pile up.
For scans and tables, conversion to text is added, and we calculate that time separately — more honest than promising a single timeframe for all input data types.
A Class of Failures That Stops Occurring
Manual field transfer produces mixed-up details, lost numbers, dates in the wrong format. These are not typos but discrepancies between documents and accounting records, which surface during reconciliation.
After implementation, they go away systematically: the parsing rule is the same for all files, and each value stores a link to the source text fragment. A discrepancy is visible immediately, not a month later.
Approaches to Annotation and Training: Comparing Modes
The training mode is chosen not by taste but by the state of the data: whether a ready corpus exists, how many entity types need to be distinguished, and how rare they are.
The first step is to fix the target metric on a holdout set; otherwise there will be nothing to compare options against.
In-house annotation makes sense when the domain is closed and expertise lives inside the team. The main risk here is consistency: annotators diverge on rare entities, and the model learns from contradictory examples.
Fine-tuning a ready-made system delivers the first working result fastest, but it inherits someone else's class schema and its quality ceiling.
Four Modes and Their Limits
| Option | When It Fits | Limitations |
|---|---|---|
| In-house annotation | Narrow domain, in-house experts, small volumes | Disagreements between annotators, timelines stretch, guidelines are hard to keep stable |
| Contractor annotation | Volume needed under tight deadlines, entity schema already described | Requires acceptance on a control sample and ongoing quality control |
| Fine-tuning a ready-made system | Little data, entity schema close to standard | Someone else's class schema, rare entities are distinguished worse, metric ceiling is lower |
| Training from scratch | Own entity schema, domain far from general corpora, data accumulates | Requires a volume of labeled examples, longer to first result |
In practice, modes are combined: they start with fine-tuning to close the task and in parallel accumulate their own annotation for training from scratch. We break down which scheme pays off in your case.
Stages of NER Model Training: From Data Audit to Handover
The technical audience is usually interested not in “we trained a model” but in which checkpoints show that it will withstand production texts.
That is why we tie the model training stages to measurable artifacts: corpus, entity schema, annotation consistency, metrics on a holdout set.
How the Work Proceeds
- Corpus audit. We review real documents and calculate the share of noise, duplicates, and language inclusions. The output is an answer to whether there is enough data and what will need additional annotation.
- Entity schema. We fix the list of types, boundary and nesting rules, and behavior at intersections. We agree on it with you before annotation: changing the schema midway is the most disruptive.
- Annotation and consistency control. Several annotators label an overlapping fragment; discrepancies are reviewed on disputed examples. We set the agreement threshold in advance.
- Training and threshold tuning. The model trains on the labeled part; the confidence of predictions is calibrated separately so that noise does not enter the result.
- Holdout validation. We calculate recall and precision for each entity type, not a single averaged number. We review errors by class and return them to annotation.
- Integration. We deliver the model as a separate service with a clear interface, connect it to your search or document processing, and monitor response time.
- Handover. Entity schema, annotation guidelines, retraining scripts, and the procedure for annotating new examples — all in your hands.
What We Hand Over: Artifacts and Procedures
A model becomes outdated faster than the domain changes: new entity types appear, the input data format changes, and annotator corrections accumulate.
That is why, together with the trained system, we hand over everything needed for its independent maintenance — so that fine-tuning and acceptance do not depend on our availability.
- Trained system — model weights, prediction service, configuration of confidence thresholds, and an environment with pinned dependencies.
- Entity schema — a list of types with rules for boundaries, nesting, and priorities at intersections.
- Annotation guidelines — a set of rules for annotators with reviews of disputed cases and examples of “this is correct / this is not.”
- Validation example set — a holdout set with reference annotation for acceptance runs and post-fine-tuning validation.
- Update and retraining procedure — frequency, trigger criteria, and the share of corrections after which training is mandatory.
- Evaluation scripts — precision and recall calculation for each entity type with error analysis by class.
- Deployment documentation — component composition, environment parameters, update procedure, and rollback to the previous version.
During acceptance, you run the validation examples yourself and compare the result with the reference. Handover is complete when the metrics match and your team has repeated the run without our help.
Case Study: Processing Incoming Requests with Data Extraction
A service company received requests by email, through a website form, and in messengers — a stream of about 4,000 per month.
Operators pulled the data for a request (contract number, site address, service type, desired date) out of the text by hand: they copied it, checked it against the database, and transferred it into the accounting system.
Before Implementation
Manual processing took 6 out of 10 requests — all where there was no ready-made form with fields. On average, one email took 8 minutes; in peak weeks the queue reached 2–3 hours, and some requests were lost in the correspondence.
The growing stream was covered by hiring new operators, and quality suffered: different employees interpreted the same field differently, which caused upstream systems to receive discrepancies in reference data.
What We Changed and What Resulted
We assembled a corpus of 12,000 real requests and labeled entities in Label Studio: five field types plus a separate class for contradictory mentions.
On this annotation, we trained an NER model and wrapped it in an extraction service — text comes from a queue, the model returns fields along with a confidence score, and the cutoff threshold is set so that with low confidence the request goes to an operator with disputed spots highlighted.
After implementation, the share of manual processing fell from 62% to 17%, and average processing time — from 8 minutes to 40 seconds. Key field extraction holds at 0.93 by F1, and the remaining 17% are mainly composite requests with embedded tables and scans.
New field types were added by relabeling several thousand examples, without rewriting the request intake logic.
How Does Fine-Tuning Differ from Training from Scratch?
Fine-tuning is updating the weights of a ready-made model already trained on large language corpora: we remove the output layer for your set of entity classes and retrain it on your annotation.
You only need to label what is not in open data — internal nomenclatures, rare formulations, industry vocabulary. For an entity recognition model, this is usually a few thousand short fragments in BIO annotation, and the first working version appears within weeks.
Training from scratch is required when the domain is far from ordinary language, there are many custom classes, and there are no ready-made solutions for them.
Then preparation on an unlabeled corpus, dictionary collection, and annotation from scratch according to the guidelines are added: the volume of labeled data grows by an order of magnitude, and project timelines are measured in quarters.
Open libraries such as spaCy provide a starting point but do not replace your annotation and quality validation for each entity separately.
| Option | When It Fits | Limitations |
|---|---|---|
| Fine-tuning a ready-made model | Standard entities, hundreds to thousands of labeled examples | Requires checking for divergence from the previous corpus |
| Fine-tuning plus rules and dictionaries | Precision on reference data, SKUs, identifiers | Rules have to be maintained manually |
| Training from scratch | Unique domain and custom classes | Full annotation cycle, long time to the first metric |
What this means for timelines: fine-tuning is a short annotation cycle and two or three training iterations; training from scratch is a separate project with data preparation.
In both cases, we first look at your corpus and estimate how many examples are actually needed, rather than naming a volume blindly.
Technical Doubts Before Project Start
Before a project starts, the same questions almost always come up: will there be enough data, what to do without ready-made annotation, how to measure quality, and who is responsible for the result. Below are short answers to them, to remove doubts before the estimate.
We have little data — will that be enough for an NER model?
Volume is measured not in lines but in the number of entity mentions: for a set of 5–10 types, a few thousand labeled mentions per class is usually enough.
If there is less material, we add fine-tuning on open corpora, augmentation, and active learning — the model itself points to the examples that will provide maximum gain. We check sufficiency using the learning curve on a trial set before the start.
There is no annotation at all — is that a blocker?
No, but we start not with the model but with the entity schema: we describe the types, boundaries, and disputed cases in the annotation guidelines. Then we label a trial sample and measure agreement between people — if it is low, we correct the guidelines, not the neural network. We label the main corpus in iterations while training and validating the model in parallel.
How will we know the quality is sufficient?
We calculate precision, recall, and F1 metrics for each entity type separately: an overall score masks a failure on a rare class, which occurs most often in the document. We fix target values in writing before training and validate them on a holdout set the model has not seen. Additionally, we run random examples from your live stream.
Who is responsible for the result?
Responsibility is divided transparently: we are responsible for annotation, training, measurements, and reproducibility of the result; the client is responsible for access to data and domain expertise.
If target metrics are unattainable with the available data, we say so before the main stage begins, not after handover. We describe acceptance conditions and the method for rechecking quality in advance.
Let's Discuss the Task and Come Back with an Estimate
To come back with an estimate and work plan rather than general timelines, we need the context of the task. An NER model is trained on your texts and your entity schema, so the input data for an estimate is first of all the corpus and an exact list of what the model must extract.
What to send to start the review:
- 300–500 typical documents or records in their original form, including inconvenient cases: scans, tables, short messages;
- a list of entity types with example values and boundaries — where the organization ends and the department begins;
- 20–30 disputed examples where manual review or rule-based search fails;
- a description of the current process: who extracts entities now, how, and how much time it takes;
- quality requirements: which entity types are critical, what may be missed, and what may not;
- environment constraints: an isolated environment without external requests or the cloud, which languages and encodings occur;
- output format: JSON, table, export to your accounting system.
The starting order is as follows: we review the data, fix the annotation schema and rules for disputed cases, and estimate the corpus volume for training.
We come back with a stage-by-stage plan — collection and annotation, training and confidence threshold selection, measurement of precision and recall on a holdout set, implementation, and post-launch monitoring. The plan relies on your real corpus, so the timelines and acceptance metrics in it are verifiable.
Describe the task — we will come back with an estimate and a work plan.
