Development of an AI System for Medical Image Analysis

Developing AI for medicine: where manual image review breaks

The flow of studies grows faster than the number of specialists ready to describe them.

During a shift, a physician reviews dozens of series, and each series is hundreds of slices in the standard DICOM format: they need to be scrolled through, compared with the previous scan, and the conclusion fitted into a report.

Department throughput is limited not by scanner quality, but by person-hours and sustained attention.

Here is where the diagnostic bottleneck lies. By the end of a shift, the time per study grows, and small formations on a homogeneous background are harder to see — the share of discrepancies between two independent descriptions in complex cases is measured in tens of percent.

Manual image review scales poorly: increased flow is handled by new positions, not by reconfiguring the process.

Developing AI for medicine addresses this as a preliminary pass, not as a replacement for the physician. The model marks suspicious areas, calculates volumes, and prioritizes the queue — studies with findings move up, norm moves down.

We evaluate medical image analysis on the department's own holdout set: what matters is sensitivity on the target class of findings and the false-positive rate, not a metric on a public dataset.

Next, we break down how this is addressed — from data preparation and labeling to integration into the current description process.

Speed and repeatability of study review

We verify the effect of implementation not with a single number, but with a set of metrics: time from a series entering the flow to a signed report, share of studies going to repeat review, variability of assessments between specialists, and class of missed findings.

We capture each metric on the same flow before changes — otherwise the improvement cannot be confirmed, and the difference is attributed to natural flow variability.

Time to report is calculated as median and 90th percentile, not average: the average hides the tail, when complex cases accumulate by the end of a shift and roll over to the next day.

If under peak load it is specifically the tail that grows while the median holds, the bottleneck is in queue prioritization, not in the review itself, and it is treated by the order in which studies are served, not by additional model training.

The share of studies going to repeat review by a second specialist is an indicator of how much the preliminary review narrows the focus area.

Its decrease means the system marks debatable cases itself and the physician looks in a targeted way, rather than rechecking everything; at the same time, the metric itself should not fall to zero — a complete absence of repeats indicates the confidence threshold is set too softly.

Variability of assessments between specialists is measured by key finding classes: agreement on them is more important than overall accuracy, because it determines whether the report will be the same for two physicians on the same image.

We analyze the class of missed findings separately — critical ones are unacceptable and must go for confirmation, borderline ones may be sent for additional labeling rather than included in the report.

Separately, we look at the throughput of the environment: how many studies per hour it handles without growth in waiting time and what happens to the metrics when the flow doubles.

Repeatability is calculated only on a stable segment — if the queue fluctuates, any figures on review quality are unreliable.

Deployment formats: cloud environment, local, hybrid

Select the environment based on patient data requirements: where studies are allowed to be stored, who has access to them, and whether leaving the protected network is allowed.

This determines the integration scheme with the clinic's information system and the model update procedure.

A hybrid environment usually becomes the default compromise: de-identification and image review run next to the storage, while heavy computations are moved to where resources can be scaled for the task.

The price for this is model version discipline and an explicit list of fields allowed to be transmitted outside.

Transition between environments is planned in advance: the input format of studies and the de-identification layer remain unchanged, so changing placement does not require rewriting the integration with the clinic. Next, we break down what is included in the work for each option.

Stages: from archive audit to handover to operation

The order of work here is not a formality: the quality ceiling of the model is set by the data, not the architecture, so we start with an audit of the study archive and only then choose the scheme.

Otherwise, it is easy to build a solution with beautiful metrics on its own dataset that gets in the physician's way in the flow.

1. Audit of the study archive: volume by modality, share of labeled series, duplicates and repeats, class imbalance, discrepancies in reports by different physicians for the same case.
2. Solution scheme: where the model runs, what data remains in the clinic environment, what goes to the physician in the response, and who makes the final decision on the case.
3. Data preparation and training: bringing series to a unified form, de-identification, additional labeling by expert physicians, configuring finding classes for the real flow profile.
4. Validation on a holdout set: we set aside part of the studies before training begins and do not show them to the model — we calculate the share of misses and false positives separately for each class.
5. Setting the trigger threshold: we select it so that rare findings are not lost and similar studies do not overload the physician with extra marks.
6. Integration: connection with the study storage and the physician's worklist, response format, behavior when the service is unavailable — queue and reprocessing.
7. Staff training and handover to operation: review of debatable cases with physicians, model update regulations, monitoring the share of discrepancies with reports, and documentation handover.

After handover, a support period remains: we see how the model behaves on new studies and retrain it when equipment or imaging protocols change.

Documentation and artifact package after implementation

Implementation is considered complete not when the model produces a report on a demo image, but when the customer can operate the system without us: change thresholds, handle incidents, connect a new room, and explain to the physician why the solution triggered the way it did.

Therefore we hand over not a 'trained model,' but a package where every decision is documented and reproducible. Below is what physically remains in your hands.

• Integration scheme — how the system is embedded in the clinic environment: which modules exchange with what, in what format images and reports are transmitted, what happens when the archive or study queue is unavailable.
• Access policies — roles and visibility scopes (physician, laboratory technician, administrator, service engineer), de-identification rules, as well as logging of every access to a study and data export.
• Model descriptions — a card for each model: what data it was trained on, which finding classes and imaging types it covers, where the applicability boundaries are, and on what set quality control was calculated.
• Model update regulations — when and by what trigger retraining is launched, who approves the rollout, how the new version is validated on a holdout set, and how to roll back to the previous one without stopping the department.
• Documentation for physicians — a short guide to the interface, interpretation of model confidence, the procedure for confirming or rejecting a finding, and actions for an uncertain result.
• Configurations and thresholds — sensitivity parameters exposed externally for the department profile, so that editing does not require rebuilding and revalidating the entire environment.
• Operational package — monitoring, list of observed signals, failure handling procedure, responsible contacts, and change log with version history.

The package is designed for your engineer and head of department to read, not the solution author. Describe the task — we will return with an estimate and plan, including the documentation composition for your environment.

Case: medical study flow before and after

The pilot was launched not on a demonstration sample, but on a live flow: about 1,100 studies per day from four machines. All of them went into a common queue, and the duty specialist manually decided what to review first.

Urgent findings waited up to 40 minutes for review, and 31% of studies were returned for revision due to incomplete descriptions.

Metrics before and after

Sorting is based on preprocessed series in DICOM format: the model separates studies without signs of pathology, marks suspicious ones, and moves them to the front of the queue.

The physician receives not a general feed, but a prioritized list where what cannot be postponed comes first.

We changed not only the model: we rewrote routing rules, added a confidence threshold and a decision log, so that every automatic report could be checked.

The log made it possible to analyze false positives during the first week and raise the share of automatic sorting from 58% to 72%. Control remained with the physician — the system suggests priority, the human makes the decision.

How does physician support differ from full automation?

The modes differ not in model accuracy, but in who forms the report and who bears responsibility for an error. Between study sorting and full diagnostic automation lies a different risk class and a different volume of evidence that must be collected before implementation.

In sorting mode, the model processes the study flow in DICOM format, estimates the probability of a critical finding, and moves the case up in the queue. The physician views the same image, just earlier — an error changes the review order, but not the diagnosis.

This is the cheapest mode to approve, and it also delivers the main effect: time to review critical cases is reduced severalfold.

Physician decision support is already a hint: lesion contours, an attention heatmap, probability by class. The physician confirms or rejects it, and responsibility for the report remains with them.

Two metrics are critical here — sensitivity on the target pathology and the false-positive rate; the threshold is selected for the specific clinic, otherwise the physician stops trusting the markings.

Full diagnostic automation requires the system to issue a verdict and refuse it on uncertain cases, passing them to a physician.

Responsibility for the report shifts to the manufacturer, and the system itself becomes a medical device — with trials and a limitation of the scope of application to one or two nosologies.

In practice, the working scheme is almost always mixed: sorting on the flow, hints on debatable cases, automation on a narrow class.

Technical doubts and limitations of the solution

The quality of the solution rests not on the model architecture, but on the data: labeling, coverage of rare findings, and behavior on images unlike the training set. We close these areas before development, not after a failure at acceptance.

Labeling follows one rule: every study is reviewed by two independent physicians, discrepancies go to a third for arbitration, and only the agreed result enters training.

The share of debatable cases is a control metric: where the labeling instruction allows different interpretations, it is refined.

Rare pathologies and nonstandard images do not dissolve into the general mass.

Classes with a small number of examples receive separate control, and images with artifacts or different positioning are included in the validation set in advance — otherwise the model confidently errs precisely on them.

There are few rare pathologies — won't the model simply not learn them?

For them, there is separate example selection and separate completeness validation for each class, rather than one common metric across all findings.

If there are physically insufficient examples, the model classifies the case as requiring physician attention: we consider a missed rare pathology a worse outcome than an extra referral for review.

Replaced the machine or changed positioning — does everything need to be redone?

No. We fix the dataset and model version, so rollback to the previous one remains available.

First we look at behavior on new images, and if quality has dropped — retraining on a small batch labeled for the new equipment, and revalidation on the previous tests so as not to lose what already worked. We do not release an update until it is confirmed on both sets.

Let's discuss the task and return with an estimate and plan

To ensure the project estimate is based on facts rather than averaged assumptions, the input requires a minimum: what task the system solves, what data already exists and in what form, what infrastructure and timeline constraints exist.

Then — a call with an engineer and a domain specialist: no presentations, only a substantive review of the task.

During the call, we clarify:

• what result is considered successful and how to measure it — share of correct findings, time to report, number of missed cases;
• what types of images are in use, whether labeling exists and who on the clinic side confirms it;
• where data is stored and processed, what requirements apply to access, de-identification, and the action log;
• what the route of an image from machine to report looks like now and at which step time is lost;
• what quality level is mandatory for the system to be allowed to work alongside a physician;
• which systems will need to be connected: study storage, workstations, reporting;
• who accepts the result stage by stage and by what deadline the first working environment is needed.

Based on the review, we send a project estimate: stage composition, implementation plan with checkpoints, requirements for data and the customer's team, a list of risks, and a method for rollback to manual mode.

After approval, we fix the first stage — work starts with data validation and a baseline model, which we refine on your examples.