Why image segmentation breaks on real data
A model that on a public dataset like COCO produces clean masks fails on entire classes on production images. The reason is usually not the architecture: in open annotation, classes are balanced, lighting is even, and objects are captured from a single viewpoint.
Real data comes from different shops and different cameras — with glare, dust, occlusions — and the average metric over the sample remains decent until a rare class drops to zero.
There are four sources of loss, and they add up.
Imbalanced classes: if a defect occupies fractions of a percent of pixels, the base loss function simply ignores it — you need class weights, targeted augmentations, and stratified validation; otherwise the model “learns” to predict the background.
Data drift between batches: a change of camera, lighting, or season shifts the distribution of colors and contrast, and accuracy drops on new deliveries, although on a holdout set from a previous batch everything is stable.
Next is annotation quality and noise in images. Different annotators interpret the defect boundary differently, polygons “wander” by several pixels, and some objects are missed entirely — the model reproduces this uncertainty as error.
Sensor noise, overexposure from a lamp, and motion blur add false positives on textures. This hits hardest on errors at object boundaries: the mask formally exists, but the contour is blurred, and any area or linear-size metric is calculated incorrectly.
We discuss how this is addressed: sample stratification, consensus annotation, drift control, and contour post-processing.
Measurable effect: accuracy, pipeline speed, manual review
What changes first is the share of frames that a human must review visually. Before implementation, a mask is built, but the confidence threshold is set with a margin, so a noticeable portion of the stream goes to review, including obviously empty areas.
After calibrating the threshold on a labeled control set, only the disputed class remains for manual inspection: falling outside the confidence interval, rare objects, images where the mask touches the boundary of the area of interest.
The share of manual review drops from tens of percent to single digits, and we separately monitor that misses do not grow at the same time.
The second metric is quality stability on new data. We look at the result not only on a holdout set but also on batches captured under different conditions: different lighting, a different camera, a shifted angle, a different density of objects in the frame.
If the spread between batches fits within the agreed tolerance, the pipeline is ready for the stream; if not, we add examples of the problematic class, retrain, and recheck again rather than manually tweaking the threshold.
The speed of processing a batch is determined not only by the model itself but also by the wrapper around it: preprocessing, slicing large images into fragments, mask aggregation, and boundary post-processing.
Here the gain comes from eliminating unnecessary passes — for example, a lightweight model filters out empty fragments, and the heavy model runs only where there is something to segment.
A batch that used to take hours is processed in minutes, and the time per frame stops depending on how busy it is.
Repeatability of the result is what the customer checks first during acceptance.
The same image with the same set of weights and settings produces the same mask, and each model version and its parameters are recorded in a log: you can see what processed a specific frame and what will change if you roll back.
Without this, metrics mean nothing — an improvement on new data cannot be distinguished from random variance. Therefore we deliver accuracy, speed, and manual review share figures together with a reproducible run protocol, not as a separate report.
Image segmentation modes: per-pixel, object, panoptic
The scene parsing mode fixes three things at once: the model architecture, the annotation format, and the acceptance metric. Chosen for a fashionable architecture rather than for the task, it forces you to re-annotate the corpus and retrain the model. Therefore we determine the mode before the first line of annotation.
| Mode | What goes into the mask | When it fits | Limitations |
|---|---|---|---|
| Per-pixel classification | each pixel is assigned to a class; objects of the same class are merged into one region | area and defect control: share of damaged surface, vegetation zones | does not separate objects standing right next to each other — they merge into one region |
| Instance segmentation | a separate mask for each instance, including within one class | counting items: parts on a conveyor, cells, vehicles, product units | more annotator time per frame; dense scenes with occlusion are harder |
| Panoptic scene parsing | all objects separately plus background as a separate class | autonomous driving, robotics, street parsing where both background and every object matter | the most expensive annotation corpus; an error on any level of classes pulls the rest down |
The class scheme is derived from the mode: classes do not overlap in meaning; rare states are better separated than dumped into “other.” The modes are data-compatible — per-pixel is assembled by merging instance masks by class, and panoptic adds a background layer. Expanding the task does not require annotating the corpus from scratch.
Implementation stages: from data audit to model handover
The order of stages here is not a formality. If you skip the data audit or fix the class scheme only after the first annotation, errors will surface on the trained model, and some masks will have to be discarded and re-annotated.
Therefore the work proceeds from data review to handover, and at each step there is a verifiable artifact.
- Dataset audit. We look at what has already been captured and annotated: scenario coverage, duplicates, frames without objects, class imbalance. The output is a report with a decision on which scenes to capture additionally and which annotation to redo.
- Class scheme and annotation rules. We fix the class hierarchy, what counts as a separate object, and how to annotate occlusions and boundaries. We write the rules down in text: without them, different annotators will produce different masks on the same frame.
- Training loop assembly. Training with dataset expansion for real shooting conditions — lighting, angle, partial occlusions. We version data and weights so that any iteration can be compared with the previous one.
- Validation on a holdout set. We calculate metrics for each class separately, not an average across the model: a rare class must not dissolve into overall accuracy.
- Validation on real batches. We run the solution on frames from production cameras that the model has not seen, analyze errors, and add difficult cases back into training.
- Solution handover. We provide weights, the class scheme configuration, retraining instructions, and metrics on the control set — the team carries the model forward on its own.
Handover artifacts: configurations, class schemes, regulations
While the solution runs in our infrastructure, its behavior depends on the people who built it. Handover is the moment when everything that lived in the engineer’s head becomes files in your repository. Below is the minimum set without which maintenance turns into dependence on the contractor.
- Pipeline configuration: parameter files for preprocessing, scaling, frame distortions, and model trigger thresholds — with an explanation of which values were changed for the task and why.
- Class scheme: a mask dictionary describing what counts as an object boundary and what counts as background, and how disputed cases at the intersection of classes are interpreted.
- Annotation instructions: annotation rules with examples of difficult frames, acceptance criteria, and a checklist for new annotators.
- Retraining regulation: when retraining is launched, on what data, and how it is verified that quality has not dropped for rare classes.
- Control frame set: reference masks that are run before publishing each new model version.
- Solution documentation: pipeline diagram, connection points to your systems, and the procedure for rolling back to a previous version.
- Experiment log: which approaches were tested and why they were rejected, so that this path is not walked again.
Such a set allows your team to change the model without us: edit annotations, retrain on new frames, and roll back a version. Describe the task — we will come back with an estimate and a handover plan.
Case study: line rejection — metrics before and after implementation
Before automation, line quality control relied on people: an inspector removed every twentieth unit from the conveyor and examined it manually.
By the third hour of the shift, attention dropped, and glue overflows, shifted labels, and seam tears on dark packaging slipped through — a defective batch went to the customer, followed by a return and re-sorting of the entire batch.
What was changed
At the line output, a camera was installed and a segmentation model was trained on annotations of real images (PyTorch).
Semantic segmentation separates packaging from the background and the conveyor belt; instance segmentation outlines the boundaries of each defect, so a glare on film is not confused with a seam tear.
The trigger threshold was selected from the error curve: it was more important not to miss a defect, but also not to stop the line with false positives. Defect rejection happens directly in the stream, and the reject discharge is synchronized with the diverting mechanism.
Metrics before and after
- Reduction of misses: about 8% of defects reached the consumer with manual sampling — it became less than 1% on a holdout set of images.
- False positives: 4% → 1.5% after threshold calibration and replacing the lighting with diffused lighting.
- Coverage: instead of one image out of twenty, 100% of the stream is checked, and conveyor speed does not decrease.
- Changeover to a new packaging size: retraining on several hundred annotated images instead of rewriting inspection rules.
How is semantic segmentation different from instance segmentation?
Semantic segmentation assigns a class to every pixel but does not distinguish instances: three cars in a frame merge into one blob of the “car” class.
Instance segmentation adds a second level — an instance identifier — so neighboring objects of the same class remain separate and can be counted. The difference manifests exactly at object boundaries.
In semantic segmentation, a pixel at the junction of two classes is a label conflict resolved by class priority or arbitration by model confidence.
In the instance segmentation approach, the same point solves a different problem: which instance it belongs to, and here mask morphology, contour smoothing, and overlapping mask suppression come into play.
The same boundary error produces a different defect class: a blurred edge versus merged or split objects.
This leads to a requirement for the class scheme. For instance segmentation, classes must be flat and mutually exclusive, and physical overlaps — a person on a motorcycle, a wire in front of a facade — are resolved by layers and drawing order.
Intersecting classes in semantic segmentation coexist peacefully, but each overlap must be described by a priority rule: otherwise neighboring annotators interpret the junction differently, and the disagreement between them directly eats into model accuracy.
The task dictates the choice. If you need the area of a class — road, vegetation, coating defect — we take semantic segmentation: annotation is cheaper, and training is more robust to rare classes.
If you need the count, size, and trajectory of each object — we take instance segmentation. A combined option stores both layers in one dataset, for example in COCO format: background goes as semantic segmentation, and objects go as masks with identifiers.
Technical concerns: limitations, support, retraining
Concerns before starting rarely concern the architecture — more often they are about data and operations: whether there are enough examples for a rare class, what to do with already accumulated annotations under a different scheme, and who retrains the model when new images appear. Below are direct answers to three such questions.
The general principle: we fix data requirements before annotation — minimum object area, occlusion rules, and how to mark ambiguous frames. Then disputed cases are decided by the protocol, not by the taste of a particular annotator.
A rare class — just a few objects in the entire dataset. Will the model even see it?
We split the sample by classes, not by whole frames, and add a rare-class weight in the loss function during training.
We set confidence for it with a separate threshold, and we calculate metrics for each class rather than an average number: the overall share of correct decisions masks failure on a rare class.
If there are fewer than a few dozen objects, it is more honest to prototype first — sometimes it is cheaper to find the class with a post-processing rule than to collect data for it.
Annotation already exists, but the scheme is different
We transfer the scheme programmatically: convert contours into masks, separate nested and intersecting regions by class priority, and discard objects that are too small by area threshold.
If your annotation was done with rectangles, we restore some masks with model pre-annotation and finish them by hand — this is faster than starting from scratch.
After conversion, we run a check: share of objects without a class, conflicting labels, frames without a single region.
You will have to retrain on your own — what is needed for that
We hand over a reproducible training and validation scenario: configuration, dataset splits, versioning of annotations and weights, so that any result can be rolled back.
Retraining is launched on new data with a mix of old data — otherwise the model loses previously learned classes.
Solution support comes down to monitoring the share of uncertain predictions and input data drift: growth in that share is a signal that it is time to add annotations and retrain.
Let's discuss the task: what to send for an accurate estimate
The accuracy of an estimate in segmentation rests not on the sample size but on how unambiguously the classes, edge cases, and required quality are described. Below is what to send so that we come back with a stage plan, timelines, and a class scheme rather than general wording.
- Task statement — what should be the output: one mask per class, separate objects, or annotation of the entire scene, and how the result is used further.
- Input data — 20–50 images in their original form, including the most inconvenient ones: low light, occlusions, noise, rare classes.
- Annotation — whether it exists, in what format (PNG masks, polygons, a ready-made dataset), who annotated it, and how consistent the disputed objects are.
- Class scheme — a list of classes, their hierarchy, and rules for ambiguous cases: occlusion, partial visibility, objects at the frame boundary.
- Quality requirements — an acceptable metric (IoU, Dice) and the classes where boundary accuracy matters and where a rough contour is sufficient.
- Deployment conditions — where the model runs: server, camera, mobile device; how many frames per day and what processing time is acceptable.
- Current process — how the task is solved now and at what volume the manual stage stops coping.
From this, a work plan is formed: data preparation, training, validation on a holdout set, and implementation into the current process.
Timeline estimates and the class scheme come together with the plan — they show the boundaries of the stages and what we consider the result at each of them.
