Why fact extraction from text breaks on real documents
On ten test documents, extracting facts from text looks like a solved task, but on a stream of a couple thousand, half the fields are empty or filled with garbage.
The reason is that real documents are unstructured: one counterparty lays out an invoice as a table, another as continuous text, a third sends a skewed scan.
The template for which the rules were configured occurs in at best a third of cases. Scans add their own class of extraction errors: recognition confuses digits in amounts and numbers, loses lines in narrow columns, merges adjacent cells.
On such input, the system produces incorrect values confidently — and that is worse than an empty field, because the error is not visible to the eye.
Nested tables collapse into a flat list, and the relationship between the line item, its quantity, and the total is lost — the output is a set of numbers without an address.
Synonyms finish off dictionary rules: "taxpayer number", "TIN", "TIN/KPP" are one field in three spellings, and each new wording requires an edit.
When failures accumulate, the team returns to manual data transfer: an operator checks fields by eye and carries them into the accounting system. Accuracy is restored, but throughput hits the ceiling of people, and errors move into fields that no one checks twice.
Next we break down how this is closed — with control at every step of document parsing, not only at the output.
Effect of automatic markup: what changes in the process
Markup ceases to be manual where a field is extracted unambiguously: details, dates, amounts, contract numbers, references to parties. The working scheme is hybrid: deterministic rules and dictionaries for regular fields, a model for free-form wording.
Such a pipeline keeps extraction accuracy at a level checked by sampling, not by exhaustive review, and the main metric becomes the share of documents processed without manual parsing.
Document processing speed is set not by the model but by queue organization: the batch is parallelized across documents, heavy pages go to a separate stream, a failure on one file does not stop the others.
A batch of several thousand pages passes in hours instead of weeks, and a re-run after editing rules takes minutes — you can change logic and immediately see the effect.
Data quality control is built on checks, not trust in the model: schema validation, cross-checking of related fields, a confidence threshold below which a record goes to verification.
Additionally, we maintain a reference set and run every change against it — regressions are caught before export, so reduced manual work is not bought by falling accuracy.
Whole classes of failures are also removed: format discrepancies across suppliers, fields lost during copying, duplicates of one entity under different names.
We build the data quality approach according to DAMA-DMBOK practice — with a field owner, a validation rule, and an edit log, so the markup result can be explained and rolled back.
Comparison of parsing modes: rules, models, hybrid
Parsing mode is chosen not by taste but by data structure. Where a value is formally defined — contract number, date, code, amount in words — rules win; where meaning depends on context — transaction party, termination condition, jurisdiction clause — a model is indispensable.
The extremes are equally costly: dictionaries break on the first rephrasing, while a model without validation produces a plausible but incorrect value, and there is no way to distinguish it from a correct one.
| Option | When it fits | Limitations |
|---|---|---|
| Rules and dictionaries | Fields with a fixed pattern: details, dates, codes, numbers | Breaks on typos and reorderings; dictionary grows faster than coverage; homonymy is resolved manually |
| Language model without fine-tuning | Heterogeneous texts, contextual relationships, a stream of new formulations | Unstable response format, misses on numbers and negations; response schema control and a confidence threshold are needed |
| Fine-tuning on a labeled corpus | One domain, a stable set of fact types, high accuracy requirements | Requires labeling and its updating when wording drifts; a separate quality control loop |
| Hybrid scheme | Industrial stream where formal and contextual fields are mixed | Debugging is harder: the error is localized to a layer — rule, model query, or check |
Human in the verification loop closes the scheme: every fact comes with a link to the source fragment, disputed cases go to an operator based on the confidence threshold, and their corrections enter dictionaries and fine-tuning examples.
Choosing an approach is choosing where to apply effort: rules are fixed quickly, the model is fine-tuned less often, but it handles formulations that were never in the dictionary.
Implementation step by step: from corpus audit to handover
Fact extraction breaks not on the model but on the document corpus: there is no single format, the same field is called differently, and some values exist only in scans.
Therefore work goes from data audit to handover — each stage is closed with an artifact that is verified before moving to the next.
- Document corpus audit. We review a sample across all types: which fields occur, how often they are missing, where the value lies — in a table, footnote, or signature. The result is a list of fields with a realistic estimate of extraction completeness.
- Field markup schema. For each field, we define the type, acceptable recording variants, and a rule for normalization to a single form. Disputed cases are described explicitly so that manual markup and the model interpret them the same way.
- Labeling the reference sample. We manually label several hundred documents and split them into two parts: one is used for tuning, the other remains closed until validation.
- Extraction tuning. We select query formulations and rules for processing the result, checking against the labeled part. Separately, we analyze fields where the system confidently produces an incorrect format.
- Validation on a holdout sample. We run the finished pipeline on documents it has not seen and calculate, for each field, the share of correct values, misses, and false positives. Acceptance thresholds are fixed before measurements, not after.
- Handover to operation. We deliver the working pipeline, processing log, interface for resolving disputed cases, and procedures: how to connect new document types and when to revise the field schema.
Handover package: rules, schemas, and validation procedures
The handover package is assembled in parallel with development, not at the end of the project: each artifact is checked on a real document stream.
The client receives not a description of the solution but working materials by which fact extraction can be run independently or handed to another engineer without verbal retelling.
- Data schema — a list of entities and attributes extracted from text: value type, requiredness, source, normalization rules, the link between a fact and its source fragment. It shows what to do when a new fact type appears without rebuilding the entire pipeline.
- Validation procedure — how a sample for manual control is formed, what share of discrepancies is considered acceptable, and at what threshold a rule is returned for revision. Without it, quality rests on one engineer's memory.
- Extraction rules with change history — what changed, on which corpus slice it was checked, which fact classes were affected, and how to roll back to the previous setting.
- Access policies — who sees source documents, who sees only extracted facts, how roles are separated, where the access log is kept, and how personal data is processed.
- Solution documentation — architecture diagram, integration points with accounting systems, description of behavior when the model is unavailable: what goes to the queue, what is discarded, how lost items are reloaded.
- Operator instructions — how to label a disputed case, which signs indicate an extraction error, where to escalate a failure, and what is considered an incident versus a routine doubt.
- Metrics log — accuracy and completeness for each fact class before and after edits, so degradation is visible in the moment, not from user complaints.
The package is accompanied by a short handover: a walkthrough with live examples and answers to the team's questions.
The scope depends on the number of document types and fact classes — at the start we will discuss the task and come back with the artifact composition and work plan.
Case: contract parsing with quality control at the output
We were approached by a leasing operator where accounting relied on manual entry: operators parsed contracts, agreements, and bank statements page by page.
On a stream of several thousand documents, this created a queue of several days, and discrepancies in terms and amounts surfaced only during reconciliation with accounting.
Fact extraction was assembled in two layers: statements in UFEB format and text contracts were parsed from structure, scans — through recognition with anchoring to a block on the page.
The model returns each value according to the described schema — parties, number, date, amount, payment schedule, subject — and necessarily with a link to the source fragment.
Quality control loop
The result does not go directly into the accounting system. First, checks run: TIN checksum, equality of the contract amount to the schedule amount, dates within the term, reconciliation of details between the contract and the statement.
A field that fails validation or arrives with low confidence goes to the manual review queue, not into the database "as it came out." Next to it is a decision log: schema version, source fragment, editor; the archive is rechecked after rules change without losing history.
Metrics before and after
- share of documents with manual entry: 100% → 9–12%;
- batch processing time: 2–3 days → 40–60 minutes;
- errors in critical fields: 4–6% → less than 0.5%;
- repeated corpus run: reprocessing under the new schema without operators.
The effect is not only speed: every fact has a source and a check, so a disputed contract is analyzed by fragment, not by an employee's memory.
What to do if fact labeling is already done manually?
Accumulated manual labeling should not be discarded — it is the most valuable asset of the project. Operators have already recorded fact boundaries, their types, and relationships between them, that is, created a reference against which any automatic parsing can be checked.
Therefore the transition begins not with model deployment but with a revision of what is labeled: which fact types actually occur, where formulations diverge, which entities are described inconsistently.
Next, manual labeling is brought to a machine-readable form: one record — one fact, with an indication of the text fragment, type, attributes, and links to related entities.
In parallel, the schema is fixed — a closed list of types and rules for their extraction, so that different labelers do not interpret the same thing differently. Such a schema becomes both a technical specification for automation and a dictionary for validating the result.
Automation starts on ready data, not from scratch: the model or extraction rules are trained and tuned on the labeled corpus, after which their result is compared with the manual reference on a holdout part.
Discrepancies show where the schema is ambiguous and where examples are lacking — this is the revision plan. Part of the labeling is deliberately left out of training: it is needed as independent quality control with every update.
The transition itself is made gradual. Automation processes the entire stream, while operators are involved only in disputed cases — those where confidence is below the threshold or conflicting rules have triggered.
Confirmed and corrected examples return to the corpus, which reduces the share of manual work from iteration to iteration, while fact labeling quality remains measurable on the same scale.
Describe how your corpus is labeled today — we will come back with an assessment and a plan for such a transition.
Technical doubts: data quality and fine-tuning
Extraction quality rests not on model size but on the data it is tuned on.
If labeling mixes different versions of contracts, and disputed fields are labeled by eye, the model confidently produces an incorrect fact — and that is more expensive than a miss, because no one checks the error.
Therefore we perform fine-tuning on the client's target corpus, not on universal datasets: first we fix the field schema and parsing rules, then collect examples, and only then touch the weights.
The second point is a closed loop. Clients rightly do not want to send contracts and personal data to external services, so we deploy model training and inference inside the perimeter. Below are answers to what is most often clarified before the start.
Can the model be trained and run entirely in our perimeter?
Yes. Training, fine-tuning, and inference run on your infrastructure; calls to external services are not required. Weights, datasets, and logs remain inside the perimeter, and outbound export is closed at the network level.
We hand over configurations, training scripts, and a process description — the team can maintain the model itself.
What happens when the document template changes?
The layout changes — field anchors are lost and extraction completeness drops. We keep the parsing schema separate from the model: rules and templates are moved into configuration, so adapting to a new form is not retraining from scratch.
Next we add a few dozen labeled samples of the new type, fine-tune the adapter, and validate on the historical set so as not to lose quality on old documents.
How do you handle disputed cases?
Fields with low model confidence do not go to export but enter the manual review queue with the supporting fragment highlighted. The operator confirms or corrects the value, and that correction returns to the training set.
This way disputed cases gradually become typical, and the share of manual review decreases from iteration to iteration.
How do you control data quality for fine-tuning?
We label according to a written instruction; key fields are labeled independently by two performers, and discrepancies are analyzed separately.
We maintain a holdout set of examples that does not participate in training and calculate accuracy and completeness for each field, not as one overall number. If a metric drops on a specific document type, it is visible before rollout.
Let's discuss the task and come back with a schema and estimate
So that the estimate and work plan are based on facts rather than guesses, it is enough to send a sample corpus — a hundred or two real documents in the form in which they arrive in the system, and a list of fields to extract: entity types, relationships between them, acceptable value formats.
If there is reference labeling for at least fifty documents, we will calculate accuracy concretely at the start. Then we will discuss the task and come back with a schema and estimate.
- A map of fact types and relationships for your subject domain: what we consider an independent fact, what is a record attribute, and what we discard as noise.
- A processing pipeline schema: document parsing, normalization, candidate extraction by the model, validation by rules and reference data, reduction to canonical records.
- A protocol for labeling the reference and rules for resolving disputed cases, so different labelers interpret the same thing the same way.
- Metrics on your documents: completeness and accuracy for each field type, the share of records sent for manual review, behavior on atypical formats.
- A pilot stage plan with explicit boundaries: sample size, set of fact types, and the quality threshold at which the schema can be handed over to industrial operation.
- An expansion procedure: how to add new fact types and rules without rebuilding the model.
- A documentation package: configurations, flow diagrams, maintenance instructions, and the procedure for rolling back to the previous version.
Describe the task in two or three paragraphs and attach what you have at hand — for us, the quality of input data is more important than its volume.
We keep the first stage short: it verifies the achievability of the required accuracy on your documents rather than demonstrating capabilities. We will come back with an estimate and work plan, and documentation handover goes together with the result, not as a separate final act.
