Fine-tuning Computer Vision models for the customer's task

Fine-tuning Computer Vision: Where Base Models Break

A ready-made model trained on public datasets confidently holds a demo and falls apart on a live object. This is a data domain shift: different optics, different lighting, flow density, and angles that were not in the training set.

In a warehouse, the model mistakes a lamp glare for a defect; in a workshop, it mistakes a shadow from tooling for a foreign object, and false positives stream in.

The second source is labeling quality. Classes overlap in meaning, some objects are missed, and region boundaries vary by tens of pixels between different annotators.

The model honestly learns from these contradictions: accuracy on the training set grows, while on the real stream it drops. It is impossible to check this without a hold-out set collected independently of the training set.

Fine-tuning "by eye" finishes it off. They retrain all layers instead of the last ones, tune the learning rate and number of passes over the same frames, and look at the averaged metric across all classes.

A rare but critical class simply dissolves in it — the final number looks decent, while the needed defect is still missed.

Therefore, fine-tuning a model on your own data starts not with training, but with an inventory of the domain: which shooting conditions actually occur, where the boundary lies between similar classes, which class is more expensive for the customer to miss than to catch an extra one.

Next — relabeling disputed frames, splitting datasets by objects and scenes rather than randomly, and monitoring each class separately.

We break down how this is addressed in fine-tuning computer vision: what to change in the data, what in the model itself, and how to make sure it has become better, not just different.

What Fine-Tuning a Model on Your Data Gives You

A base model trained on a general dataset behaves predictably poorly on your frames: it confuses close classes, triggers on glare and reflections, and gets lost when optics change.

Fine-tuning for the task shifts the decision boundary into your subject area — not on average, but at a specific warehouse, line, or road.

Accuracy and Recall in Your Domain

We calculate model quality metrics by slices, not by an averaged set: day and night, stationary camera and manual shooting, different angles.

Accuracy and recall for rare classes on each slice show where the solution is already suitable and where error analysis and additional labeling are needed.

We build robustness to shooting conditions during data preparation — we expand the set with altered lighting, rotations, noise, and blur.

Separately, we check that the model does not fall apart on a new camera: domain shift should be held by a margin in the weights, not by threshold fitting.

Eliminated Class of False Positives

The main source of false positives is not the network, but the heuristics on top of it: rules and thresholds tuned for the first demos.

We move the logic into training: we collect "hard negatives" — glare, reflections, similar objects — and add them to the fine-tuning set. We measure the effect by the share of false positives per thousand frames — before and after.

One fine-tuned model replaces a cascade of rules, so the processing time of one image stops depending on the scenario. A regression set fixes the result: edits in the data do not roll back an already closed class of errors. We work in an open stack, PyTorch, so the model runs on your side.

Model Adaptation Methods: Comparison of Approaches

What We Compare and by Which Criteria

The choice between fine-tuning a ready model for a task (fine-tuning computer vision) and training from scratch is determined not by taste, but by data: the volume of labeling, the diversity of shooting conditions, and the cost of an error on a rare class.

We compare four approaches by three criteria — how many labeled examples are needed, how the model transfers to new optics or lighting, and what will have to be retrained when a new class appears.

When an Approach Is Definitely Unsuitable

Full fine-tuning loses meaning when there are fewer labeled frames than parameters of the solution: the model memorizes the set and falls apart at the first shift in shooting conditions.

Synthetic data does not save you where behavior on real equipment matters — the generator does not reproduce sensor artifacts and glare.

Fine-Tuning Stages: From Data Audit to Handover

Fine-tuning is not launching training, but first an answer to the question of what counts as a good result. The first steps are an audit of the dataset and fixing the metric by which the model will be accepted. The order of work below is the same for classification, detection, and segmentation.

Data Audit and Metric Definition

1. Dataset audit: volume, class balance, duplicates, contradictory frames. Separately, we count the share of rare scenarios — they are what make the main contribution to errors.
2. Acceptance criteria: before the start, we fix target indicators for each class and the acceptable error type, so that acceptance does not turn into a dispute about tastes.
3. Validation scheme: we divide data into training, validation, and hold-out parts; we collect the hold-out part from an independent source or another shooting period.
4. Labeling preparation: we refine the instruction, check consistency on a cross-check sample, and close disputed examples.
5. Training: we adapt the base model to the task, selecting the mode on the validation part without looking at the hold-out set.

Validation on the Hold-Out Set

• Run on the hold-out set and analysis of model errors: we group misses by types and shooting conditions, and return the worst groups to fine-tuning or to requirements for the camera and lighting.
• Handover: weights, preprocessing configuration, a script for reproducing metrics, and a report with acceptance criteria.

Project Artifacts: Configurations, Schemes, Regulations

The delivery package in a computer vision model fine-tuning project is not only weights.

The customer receives a model with the procedure for reproducing it and the regulations by which the solution lives in operation: it is updated, checked, and rolled back without the participation of our team. Below is what exactly is handed over and why each artifact is needed.

Model and Reproducible Pipeline

• Weights of the trained model and a model card: task, list of classes, applicability boundaries, known limitations by shooting conditions.
• Training and labeling configurations: parameters, augmentation schemes, dataset versions — a repeated run gives the same result, not a "roughly similar" one.
• Processing pipeline scheme: from frame intake to response output, with quality control points and logging of each stage.
• Data access policy: roles, storage terms, anonymization, procedure for transferring datasets to third parties.
• Documentation on metrics: how accuracy and recall are calculated for each class, on which sets, and where the reference labeling is stored.

Operation and Monitoring Regulations

• Monitoring regulations: which indicators are taken from the stream, at what frequency, who analyzes deviations, and in what form they receive the report.
• Degradation thresholds and retraining rules: at what quality drop a new iteration is launched on accumulated data and on which set of examples.
• Instructions for rolling back to the previous model version and a checklist for accepting an update before rollout to production.

Reproducibility of the result is checked during handover: we deploy the pipeline on your infrastructure, train the model using the transferred configurations, and compare metrics with the reference. A discrepancy above the agreed level is a reason not to close the stage, but to analyze the cause.

Case: Defect Control on a Production Line

On an electronic module assembly line, solder joint defects were looked for visually: a shift of four operators reviewed the stream manually.

The eye gets tired by the middle of the shift, so microdefects — insufficient solder, lead misalignment, excess solder — passed further and surfaced already during assembly of the finished product. Each such find meant disassembling the unit and an unplanned line stoppage.

We captured a labeled archive from the line: several tens of thousands of frames from cameras at control positions, including a separate sample of disputed cases that operators labeled differently.

On these examples, we fine-tuned a defect classification model for specific package types and workshop lighting — training and validation on the hold-out set were conducted in PyTorch.

Metrics Before and After

Not only the algorithm changed, but also the process: disputed frames go to the operator for additional labeling, and these examples are mixed into training once a week.

The trigger threshold is configured separately for the board type, so sensitivity does not have to be squeezed to one common value.

How Fine-Tuning Differs from Training from Scratch

Training from scratch starts with random weight initialization: the network learns basic features itself — boundaries, textures, shapes — and only then the target task.

Therefore, tens of thousands of frames per class must be labeled with balance across lighting, angles, and shooting scenes.

Fine-tuning takes a model that has already seen millions of images and adjusts the features to your subject area — at the start, several hundred correct examples of a rare class are enough.

Time to production differs in the same way: from scratch it is weeks of iterations and parallel work of annotators; with fine-tuning, the first suitable weights appear within days, and on a ready set — within hours.

We freeze the early layers, train the last blocks and the classification head with a small learning rate — the model does not lose generalization ability on neighboring classes.

The main risk is overfitting on a small set. The sign is visible by the divergence of curves: on the training set, accuracy grows; on the hold-out set, it falls or stays flat.

It is treated with augmentation, early stopping, stratified splitting, and mandatory validation on a separate set from the real camera stream, not on shuffled examples.

If a third-party model is already deployed, it does not necessarily have to be discarded.

Weights are exported to a portable format — compatibility between environments is covered by the open PyTorch ecosystem — both versions are run in parallel on the same stream, and comparison is by per-class metrics, not by average accuracy. Traffic is switched gradually, with rollback to the previous version preserved.

The choice is dictated by how close the task is to what the model has already seen: a close area means fine-tuning; a unique modality and strict domain requirements mean a hybrid, where some layers are trained from scratch.

Questions Before the Start: Data, Risks, Limitations

Data requirements depend on the class of task, not on a "magic" number of frames.

For scene classification, several hundred examples per class are enough; for detection and segmentation, thousands of labeled objects are needed — and what decides here is not volume, but diversity: angles, lighting, background, object types.

Below we have collected the doubts that most often come up before the start: about labeling, the training loop, overfitting control, and robustness to changing shooting conditions.

What to do with erroneous labeling?

This is the main source of noise: the model learns from contradictions and loses accuracy on borderline cases. We run the set through cross-checking, count disagreements between annotators, and send disputed frames to arbitration. Separately, we clean classes with blurred boundaries — those are usually what the network confuses.

Can training be done on our servers?

Yes. We deploy training and inference in your environment: data does not leave the perimeter, and access is granted by roles. If you do not have your own accelerators, we calculate equipment requirements and offer an option with anonymized export on our side.

How is overfitting checked?

We maintain hold-out and control sets captured under conditions different from the training set. The gap in quality between them is a signal that the model has memorized frames rather than features. Then we add regularization, early stopping, and expansion of the set with augmentation.

What happens when shooting conditions change?

If a new camera, light, or angle was not in the data, quality drops. We build in drift control: periodically sample frames from operation and compare their distribution with the training one. In case of shift, we fine-tune on new frames without rebuilding the solution entirely.

What We Need from You to Start and Estimate the Task

The estimate for a computer vision model fine-tuning task depends not on the volume of correspondence, but on the completeness of the input data.

The earlier we see real frames and understand where the model will work, the more accurately we will name timelines and the fewer stages will have to be redone. Below is the minimum sufficient for us to return with an estimate and work plan.

• Data examples. Real images or camera recordings — including difficult frames: low light, occlusions, blur, rare object classes.
• Labeling. Whether it exists, in what form (boxes, masks, keypoints), and who confirms disputed cases.
• Current model. What is used now, what data it was trained on, and which errors interfere most — this makes it clear whether fine-tuning is enough or a rebuild is needed.
• Acceptance criteria. Which metric is considered success: share of correct detections, misses, false alarms — and on which set of frames this is checked.
• Infrastructure limitations. Where the model will run — server, cloud, camera on site; allowable response latency and autonomy requirements.
• Access. Code repository, data storage, test bench or stream recording, access approval procedure.
• Timelines and priorities. When the first working result is needed and which object classes are critical first.

This set is sufficient to discuss the task substantively and not guess. Describe the task — we will return with an estimate and work plan: we will propose a fine-tuning scheme, the scope of validation, and the stages.