Image anomaly detection: where quality control breaks down
Machine vision in manufacturing breaks not on the model, but on the input data. The camera is mounted above a conveyor where lighting changes, metal glare and vibration occur: image anomaly detection works reliably on the bench and falls apart on the line.
The first step of the system is not the network, but capture stabilization: fixed exposure instead of auto, polarization or diffuser, fiducial marks for frame calibration.
The second bottleneck is rare defects. A real defect occurs once in hundreds or thousands of good frames, so the error rate shows nothing: a model that marks everything as good is formally accurate and useless.
Quality control on the line is built around recall for the defect class and the cost of a miss: a missed crack goes to the customer, while a false alarm stops the flow and devalues the system in the eyes of the shift.
Third is labeling. Searching for defects across frames runs into the fact that operators disagree in interpretation: a scratch "by eye" is a defect for one and normal for another.
Without a procedure and cross-checking of labels, the model learns to reproduce noise rather than signal, so part of the frames are labeled again and annotator agreement is calculated.
And most importantly: a demo is not equal to the line. A different raw material batch, supplier change, lighting wear — frame distribution drifts, and the confidence threshold selected on the prototype stops separating defects from normal.
Image anomaly detection relies not on a single training run, but on drift monitoring and retraining on new frames. Below we break down how this is addressed.
Quality metrics: recall, precision, false positives
Before start, we fix three numbers by which we accept the work: missed defect rate, false positive rate, and the decision threshold at which they are measured. The threshold is a control: we lower it, catch more defects, but the flow of frames for manual recheck grows.
Therefore, in the specification we write not "model accuracy" but target recall and acceptable false positive rate.
Overall accuracy on such a flow is misleading: with a rare defect, a model that always answers "good" will show a beautiful number and catch nothing.
We calculate recall and precision separately, and analyze decisions using a confusion matrix — it shows which defect types go into misses.
We calculate model quality metrics not as an average over the entire sample, but broken down by shift, batch, line, and lighting type. Average recall may look decent until one shift or raw material batch fails — we focus on the worst group, not the overall total.
We calibrate the threshold to the cost of error: where missing a defect is more expensive than rechecking, we lower the threshold; where the operator is overloaded, we raise it.
We log decisions together with the confidence value so the threshold can be recalculated on accumulated data without retraining the model.
To verify reproducibility, we run the pipeline on the open MVTec AD dataset, a recognized benchmark in image anomaly detection.
After launch, we calculate the same metrics on a live stream: a drop in recall for a specific batch signals data drift, and adjustment happens before defects reach the customer.
Operating modes: streaming inspection, sampling inspection, archive
System operating modes do not replace each other — they cover different line speeds and different defect characteristics. Streaming frame inspection handles all incoming material and signals an anomaly immediately; archive review searches for the cause in accumulated data.
Operator sampling inspection occupies an intermediate position: the model labels the flow, a human confirms disputed frames and refines sensitivity thresholds.
| Mode | When it fits | Limitations |
|---|---|---|
| Streaming frame inspection | High-volume flow, defect is critical already at the operation output | Sensitive to lighting and angle changes; at high sensitivity, false positive rate increases |
| Operator sampling inspection | Rare defect, incomplete labeling, start of implementation | Speed limited by human; result depends on procedures and shift qualifications |
| Archive review | Search for defect causes, hypothesis testing, review of accumulated material | Does not affect current output; requires storage and metadata — time, line, batch |
In practice, modes are combined: start with sampling inspection to collect labels and estimate the false positive rate, then enable streaming inspection, and keep archive review for investigations and quality control of the model itself. This order reduces the risk that the line stops on false positives in the very first shift.
Implementation stages: from image audit to handover to operations
The order of work is arranged so that weaknesses in data and model are exposed before launch, not at the live site. Implementation stages proceed sequentially: each is closed with an artifact that can be checked before moving to the next.
- Capture and data audit. We analyze frame flow: lighting, angles, background, defect rate in the archive. Outcome — understanding which capture conditions are missing for robust detection.
- Labeling audit. We compare labels among annotators, look for missed and disputed classes. Disagreements in anomaly labeling are the rule, not the exception, and must be found before training.
- Pipeline design. We determine where the model fits into the process, how frames enter the input, and what happens on trigger. The scheme is agreed before training starts.
- Training on examples and threshold tuning. The model trains on your frames, and the decision threshold is selected separately — it determines the balance of misses and false alarms. The threshold is chosen for the cost of error in the specific process.
- Model acceptance on a holdout set. Frames not used in training are run through the model: we record the missed anomaly rate and false positive rate. The result is documented in a protocol — the baseline for comparison after retraining.
- Handover to operations. We hand over configuration, the procedure for actions when quality drops, and retraining instructions, and train the team. After launch, a retraining loop on new frames remains.
What you keep: model, schemas, access policies, procedures
We hand over not only the model weights, but the entire environment that makes it work on your side: without it, a month after launch no one will be able to understand why the model is silent on a new defect type or why events come in batches. Below is what exactly is recorded in the handover report.
- Tuned anomaly detection model: weights, selected decision threshold, breakdown by defect classes, and validation protocol on holdout frames not used in training.
- Frame processing scheme: path from capture to event recording — including buffers, behavior on communication loss, and deduplication rules for repeated triggers.
- Access policies and roles: who sees labels, who has the right to change the sensitivity threshold, who exports the event archive; action log for every configuration change.
- Retraining procedure: frequency, requirements for new labeled material, version approval order, and rollback procedure to the previous version if quality drops.
- Labeled frame set and labeling rules, including examples of disputed cases — those where the model made the most errors.
- Regression test set: reference frames with the expected result, run on every model update.
- Documentation: deployment instructions, configuration parameter description, integration scheme with your accounting system or line.
- Monitoring protocol: which metrics we watch in operation and at what deviations we consider the model degraded.
All of this is deployed in your environment and remains with you: support is possible but not mandatory — your team's engineers can retrain the model and bring up the service independently.
Case study: packaging line — missed defects and false positives
How it was
On the packaging line, defects were caught by sampling: an inspector removed every twentieth package and examined it manually.
An incomplete seam, shifted label, dent, or foreign inclusion were found with a delay — the batch had already gone to the warehouse — and the result depended on the shift, lighting, and operator fatigue.
The missed defect rate reached 7%, and another ~4% of good product went to rejection due to disputed decisions: direct losses and disputes with the recipient.
What was done and what the result was
An industrial Basler camera with lighting and sensor synchronization was installed above the conveyor: a frame is captured as the package passes, so every unit is inspected, not a sample.
The model was trained only on reference frames of good product — no need to label the defects themselves; it is enough to show normal and move the sensitivity threshold into the line settings. Disputed frames are stored separately and refine training without stopping production.
- missed defects: 7% → 0.8%
- false positives: 4% → 0.6%
- inspection coverage: sample about 5% → 100% of packages
- changeover to a new size: full shift → about half an hour for a dozen reference frames
Defects no longer reach the warehouse, and the line does not stop for batch review: the operator reviews disputed frames selectively. The same approach transfers to other areas — camera, capture zone, and reference set change; the inspection logic remains the same.
What to do if labels already exist but are incomplete?
Incomplete labeling is a normal state of a project, not a reason to start over. Some frames are labeled, rare defects are represented by a few examples, and labels on borderline cases disagree among annotators.
A model trained on such material either misses the rare class or produces false positives on similar normal regions.
The first step is to measure labeling agreement. We re-label a subset of frames independently of the first pass and calculate the match rate for each class.
Classes with high disagreement are almost always described ambiguously: we rewrite their criteria before retraining the model, otherwise it cements the annotator's error as a pattern.
We select disputed frames separately: where the model is confident but labels are not — and vice versa, where a label exists but the model is consistently silent.
Such frames provide maximum information per unit of additional labeling, so manual effort goes into them rather than into a random sample.
We build retraining on weak labels carefully: unlabeled frames are not considered negative but excluded from error calculation; confident examples of the rare class receive greater weight.
We balance with subsampling of the dominant class and augmentations of the rare class — otherwise the few defect examples drown in normal.
We monitor per class, not by overall accuracy: we separately look at recall for the rare defect and false positive rate on normal. This shows whether additional labeling helps and where more frames are still needed.
Questions we resolve before start: capture, labeling, acceptance
Technical doubts are cheaper to resolve before start than as rework of labeling and retraining. Therefore, before the first line of code, we fix requirements for capture, sample size, and acceptance criteria — they determine what comes out.
Main risks lie not in the model but in the data: rare defects, change of raw material batch, different lighting in the shop. Below is what we most often discuss with the customer's engineers before work starts.
How many images are needed for training?
From several hundred frames per defect type — with the same optics, background, and distance as on the line. Rare classes are supplemented with synthetic data and controlled augmentation; labeling is done in CVAT with double review of disputed frames.
How to check quality if defects are rare?
We calculate recall and precision separately for each defect type on a holdout set, not an average over all frames. For rare classes, we fix the decision threshold and acceptable miss rate in advance — these numbers are used to accept the work.
What happens when lighting or batch changes?
Before acceptance, we run the model on a holdout batch and frames with different exposure: metric divergence shows the stability margin.
Integration with the line is planned around threshold filtering — the decision is made inline, and the recording is saved for review of disputed triggers.
Let's discuss the task: send frame examples and capture conditions
To discuss the task concretely, it helps to collect in advance what already exists in the work: images, capture conditions, and an understanding of what miss rate you consider acceptable.
- Frame sample: several hundred images from the line — with anomalies to catch and normal frames under the same conditions.
- Capture conditions: lighting, angle, line speed, what changes between shifts and seasons.
- Defect classes: list with a description of what is critical, what is acceptable, and where the boundary lies.
- Labeling and responsible person: who on your side confirms a defect and how much time it takes.
- Implementation point: where the signal appears — operator or reject line — and what happens after triggering.
- Hard constraints: what equipment is already installed, whether cameras exist, requirements for image storage and access.
- Current inspection: how many defects are caught now, how many pass further, what the false positive rate is.
In response, you receive an analysis of your frames, a solution design for the capture conditions, a project estimate with explicitly stated assumptions, and a staged work plan — from data preparation and labeling to model stability monitoring. If there is still little data or the task can be solved more simply, we will say so directly.
