Data annotation for computer vision: bottlenecks
A model shows validation mAP near 0.9 and confident recall on rare classes, yet in production it misses defects under new lighting and confuses similar categories.
The reason is usually not the architecture: datasets were collected by one contractor following one instruction, while production receives frames from a different camera, season, and shift.
Annotation quality on validation stays steady precisely because it lacks those edge cases where the annotator made mistakes.
Most often the discrepancy between validation and production is to blame: the training dataset was collected from one batch of equipment and in one scenario, and the test portion simply repeats its distribution.
Additionally, annotation errors accumulate — cropped or shifted boxes, missed small objects, labels on background, different class definitions by different annotators at IoU 0.3–0.5. Neither loss nor metrics catch such defects: the model learns noise as a pattern.
Industry estimates converge: up to three-quarters of time in computer vision projects goes not into training but into data preparation, pre-labeling, and rechecking — according to AI Index.
A missed annotation error costs more at the deployment stage than at the collection stage: we analyze how this is closed by process discipline, not by expanding the dataset.
What pre-labeling and quality control on the stream provide
Pre-labeling is preliminary annotation of a frame by a model before an operator sees it: a detector places boxes and classes, a tracker extends objects between adjacent frames. The operator does not draw from scratch but confirms or corrects.
We perform annotation in CVAT with our own extensions for pre-labeling and review — that is why both effects below are measured on every project rather than remaining a promise.
Pre-labeling: how many times manual work volume falls
On video, tracking removes routine object propagation between frames from the operator — what remains is boundary correction at scene changes and monitoring the appearance of new objects.
On static frames, the detector pre-fills boxes with classes, and the operator works in confirmation mode rather than annotating from a blank slate. Pre-labeling has a confidence threshold: below it, an object is sent as "uncertain"; above it, it goes into spot checking.
This reduces manual work not "on average" but predictably: the share of frames sent for full correction falls by several times, and the spread in quality between annotators narrows, because the interpretation rules are set by the model, not by each annotator in their own way.
End-to-end quality control and review of disputed frames
A frame goes through not one proofreading but a chain: automated check of geometry and classes → spot cross-check → review of disputed cases by a separate team.
Disputed frames are not "smoothed over" by majority vote but go into a queue with a reference decision — it becomes a rule for similar situations and is recorded in the instruction.
Annotation quality control produces a measurable result: the share of corrections after proofreading falls from iteration to iteration, and the model metric on a holdout set stops jumping between retrainings.
Comparison of data annotation modes: in-house staff, contractor, hybrid
The choice of annotation vendor decides not the price question but the question of object class and stream volume.
An in-house team of annotators keeps disputed classes and instruction edits under control, but handles peaks poorly: hiring and training take weeks, and people idle between projects.
Data annotation by a contractor covers peak load and brings ready-made quality control regulations, but every instruction edit goes through an approval cycle. A hybrid annotation scheme keeps the core and acceptance in-house while giving mass volume to an external team.
| Option | When it fits | Limitations |
|---|---|---|
| In-house team of annotators | Narrow classes with high cost of error, closed data, frequent instruction edits | Scales slowly, expensive to keep idle between projects |
| External team | Mass stream of similar frames, stable instruction for months | Depends on instruction quality and annotator rotation |
| Hybrid scheme with pre-labeling | Volume peak together with disputed classes, need a quick start | Requires an in-house core for review and acceptance |
Limitations by object types
Boxes and points are annotated faster and go to an external team almost without losses.
Polygons, masks, and skeletons require unified rules on boundaries and overlaps: here a contractor is connected through model pre-labeling, while disputed cases are reviewed by the in-house core.
What changes as frame volume grows
On a stream, the share of missed objects and double boxes grows by several times — quality control becomes the bottleneck, not annotation itself.
A hybrid survives this tipping point more easily: pre-labeling removes routine, spot checking catches discrepancies, and agreement between annotators shows when it is time to rewrite the instruction.
How the process is structured: audit, schema, pilot, handover
We structure the process so that annotation does not turn into a black box: at each stage there remains a verifiable artifact and a quality metric. This shows which result appears at which step.
- Audit of the source dataset. Frame composition, shooting conditions, duplicates, share of frames without target objects, corrupted exports.
- Class and attribute schema. List of classes, schema version, rules for edge cases: overlaps, partially visible objects, glare.
- Preliminary annotation by a model. The annotator manually corrects the network's draft — the pass speeds up by several times, but control remains.
- Quality control. Cross-check of disputed frames, agreement metric between annotators, review of discrepancies with instruction updates.
- Pilot batch. We process a small slice end-to-end and calculate the error rate — using it, we correct the instruction before the main volume starts.
- Acceptance and handover. Export in the format your model accepts, version registry, instructions, training for your team.
What an audit of the source dataset shows
Most often it reveals the non-obvious: some frames are duplicated, many have no objects of the required class, and annotations by different annotators contradict themselves. Without seeing this at the input, the error is scaled to the entire dataset.
How acceptance and training of the customer's team are structured
At acceptance, we spot-check the batch according to the same rules as the annotators and show discrepancies. Then we hand over documentation on the class schema and go over with your team how to accept subsequent batches independently.
Deliverable package: schemas, regulations, documentation
Annotation without documentation lives only until the first annotator change: a new person interprets classes in their own way, and the dataset starts diverging from the original schema.
That is why the deliverable package is not an archive with files but a set of documents by which the next annotation team or your engineer reproduce the same annotation without us.
Annotator instruction and rules for resolving disputed cases
The annotator instruction is a working document for a specific schema, not a general guide: for each class, the procedure, boundary examples, and typical errors with visual references are specified.
Disputed cases are not decided on the spot — the annotator flags an object, the arbiter records the decision in a journal, and it enters the next revision of the instruction.
- Annotation and attribute schema — class hierarchy, geometry types, mandatory and optional features, overlap rules, and minimum object size.
- Annotator instruction — algorithm for each class, visual references, analysis of typical errors.
- Quality control regulation — share of rechecking, agreement threshold between annotators, rule for sending a task for re-annotation.
- Rules for resolving disputed cases — escalation levels, arbiter role, journal of adopted decisions.
- Annotation documentation — export formats, class dictionary, schema version history.
- Validation set for accepting new annotators and a quality report: which classes account for the main share of errors and how the metric changes from iteration to iteration.
Case study: how a pilot reduced the share of rework at review
The client provided a stream of camera recordings: night scenes, overlaps, objects at the frame edge. Annotation was performed in CVAT by a distributed team — a twenty-page instruction, acceptance at the last level.
Consistently 16–18% of frames were returned for rework, average time per frame stayed around 40 seconds, and the dataset grew slower than the training schedule required.
What exactly was changed in the process and why it worked
Manual drawing "from scratch" was removed: a pre-annotation model trained on already accepted frames gives the annotator ready-made boxes, and they correct them. The longest stage — finding object boundaries — disappeared, and time per frame fell to 12–15 seconds.
Quality control was moved from the batch to the frame. Disputed cases — overlaps, blur, an object at the boundary — go into a separate queue with an explicitly described decision criterion, so not the entire dataset is rechecked but 6–8% of the sample.
The instruction was replaced with a short checklist with reference examples: discrepancies between annotators decreased.
Results by metrics: rework share — from 17% to 4%, time per frame — from 40 to 14 seconds, volume of repeated checks — three times less.
Annotation quality metrics — reviewer agreement and share of frames accepted on the first pass — are captured on every delivery, so growth in volume does not dilute quality.
How does pre-labeling differ from annotation from scratch?
Pre-labeling and manual annotation lead to the same result — correct boxes, masks, or keypoints on a frame — but they get there differently.
In automatic pre-labeling, the model places draft boxes, and the annotator confirms, shifts, or deletes them; in annotation from scratch, the operator draws each object themselves. The difference shows in labor costs per frame and in how quality behaves on dense scenes.
| Criterion | Pre-labeling | Manual annotation from scratch |
|---|---|---|
| Who draws the object | model, human corrects | human entirely |
| Time per frame | several times less on homogeneous classes | grows linearly with number of objects |
| Object boundaries | depend on the model, need refinement | more stable on non-standard shapes |
| Dense scenes | merges and misses, control needed | manageable, but slow |
| When it fits | data stream, large homogeneous classes | rare classes, fine boundaries, disputed cases |
In practice, the approaches are combined: pre-labeling covers the main volume, manual review covers classes with high cost of error.
The boundary is determined by measuring on a sample: if the annotator corrects less than a third of the boxes, automatic pre-labeling pays off; if more, the model drags its errors into the dataset, and on this class it is cheaper to return to annotation from scratch.
When is manual review needed? Where an object is overlapped by background or neighboring objects, where a class is visually indistinguishable from a close one, and where an error is costly: medicine, defect detection, transport accounting.
Here control is continuous, plus measurement of agreement between annotators — identical cases must receive identical labels, otherwise the model will learn the contradiction.
Technical doubts: quality, timelines, access
Before the start, the technical side raises more questions than price. Below are five doubts that most often come up during approval, and answers without general words: they rely on the acceptance regulation and on how annotation review is structured.
By what metrics do you accept annotation?
Annotator agreement on a control subsample, the share of mask intersection with the reference (IoU), and recall on rare classes. We fix thresholds in the regulation in advance: a batch does not go into work until it passes all three.
How long will annotation take?
The timeline depends on object density per frame and the number of classes. We annotate the first batch with measurement of actual output — after that, the forecast is based on real speed, not on a norm.
How do we transfer images?
There are three working schemes: export to our storage, annotation on your infrastructure, access through a secure channel with a role model. We anonymize personal data at the input, and revoke rights after handover.
We already have a partially annotated dataset
We conduct an inventory: compare the class schema, look for conflicts and duplicates, complete missing frames, and bring everything to a single format — COCO, YOLO, or Pascal VOC.
Can annotation be reused?
Yes. The class taxonomy and rules remain an asset: the next project does not start from scratch — the schema is ready, some frames come with preliminary annotation, and review is performed only on disputed zones.
Let's discuss the task: where to start and what is needed
To come back with an estimate for an annotation project rather than general formulations, we need a minimum of starting materials. The estimate rests on three quantities: how much data, by what rules to annotate it, and how to verify the result.
The more complete the input, the more accurate the annotation volume and timelines — and the fewer reworks at the delivery stage.
- Sample of raw data — 200–500 files from a real stream, not selected "successful" frames: they reveal noise, duplicates, corrupted files, and rare classes.
- List of object classes and annotation rules: what counts as a separate object, how to annotate overlaps, cropped objects, and disputed cases, what minimum object size is taken into account.
- Expected annotation volume: number of frames and objects, share of frames for continuous review, whether annotation from scratch is needed or model pre-labeling with operator completion is sufficient.
- Export requirements: directory structure, annotation format, split into datasets, and how the result will enter model training.
- Acceptance criteria and quality metrics: permissible error rate on a sample, agreement between annotators, procedure for re-annotating disputed examples and handling customer comments.
- Materials from previous iterations, if any: old annotations, correction log, known problematic classes — this removes a separate calibration cycle.
Describe the task and attach what you already have — we will come back with an estimate and work plan: stages, timelines, team composition, and quality control points at each step.
