Why automatic document classification breaks on a stream
While the incoming stream is homogeneous, automatic document classification relies on rules: sender, file name, set of keywords, regular expression on the header.
The scheme can be read by eye and fixed in minutes, and metrics on historical data look almost perfect — on a stable stream this is indeed enough.
Growth of the incoming stream breaks the main assumption — homogeneity. New counterparties arrive, scans without a text layer, migrated templates, files with two grounds at once.
A rule that looked for one line silently stops triggering, and a fixed confidence threshold passes on everything remotely similar to a familiar sample.
Then the cascade kicks in. Routing errors send an invoice into contract approval, and an act into accounting, and this is discovered a day later, when the deadline is already pressing.
Each miss adds a document to the manual sorting queue, and it becomes the processing bottleneck: the operator handles not exceptions, but the entire volume that the rules failed to catch.
We explain how this is closed: the classifier looks at content, not at formal attributes, learns on your labeled examples, and sends the document to a route together with a confidence score — disputed cases go not to a general queue, but to a specific specialist.
What classification gives in practice: speed, accuracy, risks
The effect is measured on three metrics: stream processing speed, error rate on labeled control, and volume of manual review. We calculate them on a real incoming stream, not on a showcase sample — then both peak days and atypical documents are visible.
- Processing speed. The incoming queue is processed continuously: overnight arrivals are ready by morning, and the peak is handled in minutes, not by a shift of operators.
- Error rate. For high-volume types, accuracy stays above 95%; most misses are rare types and poor scans. Text reading goes through recognition on Tesseract, so the quality of the source image remains the main limitation.
- Manual review. Only part of the documents with low model confidence goes to it: the operator sees the top hypotheses and confirms one, rather than entering details from scratch.
- Process downtime. The bottleneck shifts from processing to arrival: waiting in the queue is reduced from hours to minutes, and stuck documents are visible in monitoring immediately.
- Eliminated failures. The "wrong type" class of errors goes away: a document that got into the wrong route no longer gets stuck in approval.
- Feedback. Operator corrections are collected into labeling, the next training iteration reduces the error rate and adds new types without rewriting rules.
We define the boundaries honestly: types that the model distinguishes with low confidence we close with an explicit rule or human confirmation, not with a guess.
Approaches to processing: rules, learning on labels, hybrid
Processing starts with the decision of which signal to consider a type attribute: the form structure, stable anchors in the header, the set of details, or the text itself. As long as suppliers do not change layouts, this is enough.
As soon as the stream changes — a new counterparty arrives, the form is updated, a scanned copy appears instead of a file with a text layer — rules begin to trigger falsely, and processing quality declines unnoticed by reporting.
We do not choose the scheme in advance: we calculate metrics on an export of the customer's real documents.
If 70–80% of the stream is of the same type, it is cheaper and more reliable to close them with rules — each decision is explainable, and an error is visible in a specific condition. We give the remainder to the model.
Three schemes and the cost of maintaining them
| Approach | When it fits | Limitations |
|---|---|---|
| Rules and templates by anchors | The stream is homogeneous, forms are fixed, explainability of each decision is important | Breaks when the layout changes: form edits cause false triggers, the list of exceptions grows faster than the rules |
| Learning on labels | Document types are heterogeneous, the useful signal is content, not appearance | Requires a labeled sample and its regular replenishment; on new classes quality drops, data drift monitoring is needed |
| Hybrid | There is a stable core of forms and a long tail of non-standard documents | Two layers are versioned and debugged together; an error comes from rules, from the model, or from their priority |
How we implement incoming stream processing: stages of work
The order of work is arranged so that processing quality is measurable before launch into the production loop. Therefore we start not with the model, but with a description of the stream: which document types arrive, in what volume, and where manual work currently arises.
- Stream audit. We count volume by type, look at intake sources — email, scans, file exchange, integrations — and record the share of documents that a person processes today.
- Class scheme. We agree on a closed list of types and rules for disputed cases: a document falls into exactly one category, otherwise quality cannot be measured.
- Sample labeling. We collect a representative sample across all types, including rare and borderline ones, label it according to agreed rules, and divide it into training and holdout parts.
- Processing setup. We train the model on the labeled part, add text recognition from scans and detail extraction, and check quality separately for each type.
- Check on the holdout sample. We evaluate accuracy by class, the error matrix, and behavior on poor scans — the model has not seen this data. The error rate here is the readiness criterion.
- Confidence threshold. Documents with low confidence go to an operator, not to automatic processing; we retrain the model on their corrections.
- Solution acceptance. We agree on metrics, run the stream on real volume, and hand over the configuration, labeling rules, and quality control regulation.
What remains with you after implementation: configurations and regulations
We hand over not a "working model in your loop," but a full set of documentation that your support team can use to run the system without us. Below is what exactly remains in your hands after handover.
- Solution scheme — from the document entry point to recording the result: which stages the file goes through, where recognition quality is checked, what happens when a node fails, and where the notification goes. With it, a new engineer enters the system without parsing someone else's code.
- Type map — a registry of classes with examples and a description of the attributes by which a type differs from a neighboring one (for example, an invoice and an act with a similar header). When a new document type appears, it is clear where to add it and which rules are affected.
- Processing regulation — what to do with low-confidence responses: the threshold for sending to manual review, the operator queue, processing deadlines, routing by department.
- Instructions for operators — how to confirm and reject a result, how to mark an error so that it goes into the next labeling, and which error types are more important than others.
- Model update procedure — labeling rules for new examples, requirements for sample volume per class, the procedure for running checks on the holdout set before publication.
- Metrics and reports — the share of correct answers by type, not an overall figure: it is visible which class degrades first and when intervention is needed.
- Version log and rollback — which version is currently in operation, how it differs from the previous one, and how to return to it if quality has declined.
Such a set closes the main implementation risk: a system that only its author can support. We hand over the materials together with processing to your team — describe the task, and we will return with an estimate and a plan.
Case: incoming stream processing in an accounting system
Where the stream stood
The incoming stream of the accounting system — invoices, acts, contracts, delivery notes, scans and photos — was processed manually by operators: open the file, determine the type, sort it into processing routes.
At a volume of about 4,000 documents per day, mechanical labeling consumed up to 60% of shift time, and on peak days regulatory deadlines were missed.
What was changed in the pipeline
We assembled a sequence: text extraction via Tesseract for scans and structure parsing for "digital" files, then a classifier by document type with a confidence threshold.
Everything below the threshold is not discarded or guessed — it goes to the manual review queue with the type already prefilled, and the operator only needs to confirm or correct it.
Metrics on the same stream
Type detection accuracy increased from 78% to 96%, the share of manual review fell from 100% to 21% — and these percentages come mainly from crumpled scans and mixed batches. The average time from file arrival to sorted document decreased from 38 minutes to 2.5 minutes.
We separately analyzed the error class: incorrect routing of an invoice into the acts block previously surfaced during reconciliation a week later; now type conflicts are caught at the input.
The load per shift decreased by about four times, peak days stopped being a bottleneck, and the freed time went to disputed documents instead of mechanical sorting.
What to do if rules and selection scripts are already configured?
There is no need to disable the rules — the model runs alongside them and initially works in shadow mode: the document goes through both scenarios, and the rule's decision remains in effect while we compare results on the live stream.
On this sample, it is visible where the conditions are mistaken and where the model behaves uncertainly — and a real accuracy reserve appears, not a promised one.
Then confidence-based routing is enabled. If the model labels the class confidently, the decision is made immediately; if below the threshold, the document goes by the old conditions or to an operator.
The threshold is configured separately for each class, so for critical types the rules remain the only source of decision, and the model only learns on them.
Rules often live directly in the accounting system — 1С:Документооборот and similar loops, where every change is coordinated. Hence the transition order: first a shadow run, then one document type, then the rest, and only after that conditions are disabled one by one.
The main weakness of rules is new document types: every new template or counterparty requires another set of conditions, and support grows faster than the stream itself.
The model closes this with retraining: several hundred labeled examples of a new class, a check on a fixed set, and the class is introduced without rewriting logic. Disputed documents go into the labeling queue — it also becomes material for the next retraining.
Technical doubts: accuracy, labeling, retraining
Questions at the stage of comparing contractors repeat: how quality is measured, who labels the data, how the classifier interfaces with the accounting system, and what changes after launch. Below are answers without presentation wording.
What accuracy should be considered sufficient and on what do you measure it?
We calculate accuracy separately for each document type: in the stream, a rare class has little weight, but an error in it is the most expensive. Measurement on a holdout sample with an error matrix shows which similar forms the model confuses. We agree on the threshold before the start, and uncertain answers go to manual review.
How many documents need to be labeled and who does it?
Labeling limits the project more than code does. Several hundred examples per class are enough; we supplement rare types with transfer and synthetic data.
Labeling is done by the customer's employee who knows the domain, and we record rules and disputed cases in a decision registry.
How does the solution interface with the accounting system?
We do not rewrite the accounting system: the classifier is a separate service that takes a document from a folder, mail gateway, or bus and returns the type, confidence, and recognized fields.
Exchange is via an open interface, and a repeated call does not create a duplicate record. First a test loop, then parallel operation, then automatic posting.
What happens after launch and what does retraining look like?
The model degrades: forms, suppliers, and regulations change. We monitor the share of correct answers in the stream, collect operator corrections into a training set, and run retraining on a schedule or when a metric drops. We store model versions — rollback to the previous one takes one operation.
Let's discuss the task and return with an estimate and a plan
For the estimate to be accurate, and not taken from thin air, we need a slice of your real stream — a full export is not required, a representative sample is enough. Gather what is already at hand, and we will clarify the rest in conversation.
- 200–300 documents of each type in their original form: scans, DOCX, photos — this shows the quality of shooting and layout we will have to work with;
- a list of classes into which documents must be sorted, with examples of disputed cases — where the boundary between a "supply contract" and an "invoice" is, you determine;
- how documents arrive now: email, scanner, folder, export from the accounting system — and daily volume;
- examples of manual labeling, if it was done: it speeds up model training;
- requirements for the system response: class label, confidence level, what to do with uncertain documents;
- access restrictions: whether data must be processed inside your loop.
We will return with three things: a project estimate by stages, a work plan with team composition and timelines, and a forecast of the effect in measurable terms — the share of documents processed without a human, accuracy on your sample, processing time for one document.
Then we propose a pilot on a limited stream: one or two document types or one department, but on live data and with metrics measured before and after — the result shows whether to scale the solution to the entire volume.
Describe the task in free form and attach the sample — we will return with an estimate and a plan.
