Automatic diagnosis coding: where accuracy is lost
The chain "physician's note → ICD code" breaks not at the statistics stage but at the input.
Free text in the medical record is written for treatment, not for a classifier: abbreviations, synonyms, local designations, word order that depends on the habits of a particular department.
Then a person turns the diagnosis into a code, and they reconstruct the picture from the note, not from the examination.
Manual coding errors are weakly related to the qualification of the coding specialist.
The main sources are the multiaxial structure of ICD-10, where one diagnosis is assembled from several categories, and disputed ICD codes: borderline states, combinations of primary and concomitant disease, the choice between a more specific subcategory and an "unspecified" category.
Two different people label the same text differently — this is the industry norm, not a shortcoming.
The bottleneck of labeling is throughput.
For a code to become reproducible, the record must be normalized: expand abbreviations, separate the negation context ("excluded," "questionable," "history of") from the assertion, distinguish the primary disease and the background.
At scale, this is hours of monotonous reading, and attention drops the closer the shift gets to the end: discrepancies grow, and along with them the volume of manual returns for recoding.
Automatic diagnosis coding targets exactly these gaps, not post-hoc statistics. The model reads the original wording, proposes a ranked list of codes with confidence for each, and disputed cases go to a human for review — instead of total manual analysis.
Let's break down how this is arranged and where the boundary lies between an automatic decision and mandatory control.
What the solution delivers: processing speed, accuracy, disputed records
The pipeline does not work as "guess the code from a string" but as a conveyor: normalization of the record text, extraction of clinical entities, matching against ICD-10 categories, ranking candidate codes by confidence.
A code above the threshold is auto-confirmed; below it goes into the manual review queue. On a continuous stream, this gives record processing 4-6 times faster than manual entry.
Accuracy on auto-confirmed codes is 92-96% agreement with the coder's conclusion on a held-out set.
The remainder mainly falls on composite diagnoses: primary disease plus concomitant conditions, codes with an additional characterizing code, cases where the wording changed during treatment.
There, a single code is incomplete, so the system returns a set of codes and indicates which fragment of the record supports each one.
Disputed records — 5-8% of the stream — are not guessed at: these are incomplete formulations, local abbreviations, conflicting mentions in different fields, handwritten input.
They go to the coder with the fragment highlighted and the nearest candidates, so the decision takes seconds instead of analysis from scratch.
The coder's freed-up time goes specifically to these cases and to replenishing parsing rules, and each decision becomes a labeled example — the auto-confirmation rate grows month by month.
Operating modes: rules, learning on labels, hybrid
Modes differ not in the model's "intelligence" but in the source of the code and who is responsible for the error.
ICD coding rules give a deterministic result on typical formulations, learning on labels generalizes to the physician's free text, and the hybrid mode combines both sources. The choice of mode determines where errors will occur and how many records will go to manual code review.
How to choose a mode
| Mode | When it fits | Limitations |
|---|---|---|
| Rules and a dictionary of diagnosis synonyms | Homogeneous stream, narrow department profile, requirement to explain every code | Does not cover formulations outside the dictionary, expanding the dictionary is manual work, false positives on homonyms |
| Learning on labels | Varied formulations, there is an archive with confirmed codes | Requires a volume of labels, sensitive to a change in classifier revision, the decision is hard to explain to a physician |
| Hybrid mode | Mixed sources, high accuracy needed with moderate manual work | Two pipelines to maintain, discrepancies between them must be analyzed separately |
Hybrid mode works like this: rules and the dictionary produce a candidate with justification, the model ranks the options and marks records where its confidence is below the threshold. Such cases go to manual code review; the rest are coded automatically.
The share of records that reach a specialist is a working metric: it grows when the classifier is updated and suggests where to supplement the dictionary or labels.
Implementation stages: from record audit to handover
The work proceeds in sequential stages: each next one relies on the result of the previous, so a problem is visible at its own step, not at acceptance. Below is a typical order that we adapt to the client's pipeline and data protection requirements.
-
Audit of medical records. We take a de-identified slice for a period and look at the form in which the diagnosis arrives: free text, abbreviations, already assigned codes. The output is a data quality map and a list of classes that the system must distinguish.
-
Data processing scheme. We determine the source of records — integration with a medical information system or a scheduled export — the de-identification points, and the storage procedure. We agree on the scheme with the person responsible for personal data.
-
Rule setup. We assemble dictionaries of synonyms and abbreviations, priority rules between similar codes, and mandatory clarifying features. This fixes what the physician applies automatically and removes some discrepancies before the model.
-
Reference labeling and training. Expert physicians label the reference sample, we analyze disputed cases collegially, and the model learns on agreed data. The result of this stage is a version with fixed accuracy by class.
-
Validation on a held-out set. We run the system on records it has not seen: separately for frequent diagnoses, separately for rare and mixed formulations. We analyze discrepancies and fix either the rules or the training.
-
Handover to operation. We hand over the error analysis procedure, quality monitoring, and retraining order, train employees, and remain available for support.
Each stage is closed with an artifact — a data map, a scheme, a set of rules, a metrics report — which the client accepts and checks.
Handover package: rules, dictionaries, control procedures
The model is only part of the system. Alongside it, the client receives a package that allows automatic diagnosis coding to be run and maintained in-house, without being permanently tied to us. Below is what exactly remains in hand after handover.
- Set of coding rules. How to choose the primary diagnosis, how to code concomitant conditions and complications, what to do with unspecified formulations. The rules set the priority order and explicitly list cases where the system must pass the decision to a human.
- Mapping dictionary. A table between physician formulations and ICD-10 codes: synonyms, abbreviations, typical typos, examples from your practice. It is maintained by version — you can see what changed and when.
- Quality control procedure. What share of cases goes to sample review, how discrepancies are analyzed, at what drop in accuracy a revision of rules is triggered.
- Instructions for coders. What to do when the system returns two close variants, how to document a disputed case and where to route it.
- Integration scheme. The points where the code enters the record system, where the physician confirms or changes the result, what data is transmitted and in what form.
- Update procedure. Who enters new formulations into the dictionary, who approves edits to the rules, and how it is verified that an update has not broken already working cases.
Such a package allows changing rules for a new revision of the classifier or for the clinic's internal agreements independently — without retraining the model at every step.
Practice: record processing accuracy before and after changing the rules
Before the restructuring, diagnosis parsing rules lived inside the code: a clarification for each record template, a dictionary with duplicate categories and overlapping formulations.
At scale, accuracy on prepared examples looked decent, but on real records it dropped — the coder manually corrected about a third of episodes.
In the diagnosis coding project, we moved the rules out of the code into a separate editable set, and brought the dictionary to the hierarchy of the current WHO revision: removed duplicate categories, set priority for clarifying subcategories, and described contextual rules for combinations of diagnoses.
The model returns a code only when it relies on the record text — if there is no support, the record goes to a human for analysis rather than receiving the nearest suitable code.
Metrics before and after were calculated on the same stream. Accuracy rose to 92–96% depending on the department profile, the share of manual edits decreased by about three times, and the error class "code is not supported by the text" almost disappeared.
Validation on a held-out set — records that participated neither in rule setup nor in dictionary edits: the gap with the working sample does not exceed two points, so the growth is not overfitting to specific examples.
Rules are versioned: every change is run on the held-out set before release, and regression is visible before it reaches the coder. This is what keeps accuracy in the stream, not in the report.
What to do if some diagnoses go to manual refinement?
Automatic diagnosis coding does not have to cover one hundred percent of records — some always go to manual code refinement. The question is not how to remove the human entirely, but how many records they process and in what order.
If the remainder is not prioritized, the review queue grows faster than the gain from automation arrives.
The system's confidence threshold controls this. For each record, the model returns a code and an estimate of its own confidence: above the threshold, the record goes into the main stream without human involvement; below, it goes into the review queue.
The threshold is not a constant but a setting with a direct cost: shifting it down reduces manual refinement but increases the share of errors that will surface later; shifting it up reduces risk but gives specialists more records than they can process per shift.
Therefore the queue is built not by arrival time but by risk. Records rise to the top where alternative candidates are close in score, where a rare code or an incomplete diagnosis formulation occurs, where there is a noticeable discrepancy between the text and the selected code.
The specialist does not search for the code from scratch: they see several most probable variants with an indication of which fragment the system relied on, and confirm the choice in seconds.
This is how partial automation becomes a working mode rather than an intermediate stage.
Three quarters of records pass without a human, the remainder is distributed across shifts predictably, and the share of returns for repeat review falls — not because the model has become error-free, but because disputed cases are intercepted before export.
We set the threshold and queue rules at the design stage, together with metrics for specialist workload.
FAQ: data, integration, quality control
Before evaluation, the same set of questions comes up: where the data comes from, how the solution will fit into the working pipeline, and who is responsible for labeling quality. We answer in advance — both the architecture and the scope of work depend on this.
How can this be integrated into an existing medical system?
We do not change the core of the record system: the text of the conclusion is supplied as input, and the system returns an ICD-10 code with confidence and a reference to the fragment that influenced it.
The physician sees the suggestion in the working window and confirms or edits it — rollout can be gradual, starting with one department.
How is personal data protected?
We de-identify before it enters the model: full name, contacts, and document numbers are removed, and the link to the patient is stored separately, in the client's environment. Access to logs is role-based, and requests are logged. If necessary, processing is deployed entirely within the clinic's infrastructure.
What happens when the ICD dictionary is updated?
The dictionary is versioned: the new revision is first run on historical cases with a known correct code and only then put into operation. The previous version is retained, so rollback takes minutes rather than requiring reconfiguration.
How do you check labeling quality?
We assemble a reference sample: expert physicians label conclusions, and we analyze disputed cases collegially — this is how an agreed boundary of the correct answer emerges.
We calculate accuracy separately for rare and complex diagnosis groups, where an error is most costly, and add discrepancies to the training set.
Let's discuss the task and come back with an estimate and work plan
For the coding project estimate to be substantive rather than "based on feelings," we need a real slice of your data.
Automatic diagnosis coding can only be validated on your formulations: an export from another institution has different vocabulary, its own abbreviations, and its own field order.
Therefore the start is a discussion of the task and a corpus of records, not a description "in general terms."
What to send for the start:
- 200–500 de-identified records with diagnosis text — and, if available, with an already assigned code;
- the code dictionary you work with: full ICD-10, regional, or internal;
- 10–15 disputed cases where coders disagreed;
- the structure of record fields: what the physician fills in as mandatory and what as free text;
- requirements for the processing environment — whether external network access is allowed and how data is stored;
- the target metric: what share of records may be sent for manual review.
In response, we return an estimate of the scope of work and an implementation plan: the processing scheme, the volume of labeling and retraining, a test run on your records with accuracy by code and rejection rate, the composition of integration with your record system, and stages with acceptance points.
If, based on the test run, accuracy on rare classes turns out to be lower than stated, this will be visible before the main stage begins, not after.
