Why image classification breaks on real-world data
Most often the request sounds like this: “the model shows almost perfect accuracy on the holdout set, but in production it makes mistakes on half the objects.” We break down exactly where image classification breaks and why switching to a heavier architecture does not fix anything.
The first source of failures is dataset labeling tailored to laboratory conditions. Images were taken in one room, under one light and angle, labels were assigned by source folders, without a single rule for disputed cases.
As a result, the model learns the background and shooting artifacts, not the object feature: just change the table or lamp, and accuracy drops.
The second is class imbalance. When a rare category accounts for fractions of a percent, overall accuracy remains high simply because the model confidently predicts the frequent class.
You need to look at precision and recall for each class and at the confusion matrix; otherwise the main failure class is not visible in any summary figure.
The third is a change in shooting conditions: a different camera, lighting, season, position of the object in the frame.
The input data distribution drifts from the training one, and the model sees something other than what it was trained on — even a carefully trained ResNet or ViT does not save you here.
We break down how this is addressed: first we fix the labeling rules and relabel disputed groups, then we balance classes at the level of example selection, not just weights, and only after that we train the network.
What a trained image classification model provides
A trained image classification model turns a stream of images into a decision with a number. Model accuracy is fixed before launch on a control set — a set of frames labeled manually and not used in training. It is this, not the result on the training set, that becomes the acceptance parameter.
On the stream, the model returns a label and confidence, and the confidence threshold determines what share of decisions goes to automation.
Everything below the threshold goes to manual sorting: review becomes selective instead of continuous, and the operator handles only disputed frames.
The error rate on the control set is known in advance, so the workload per shift can be planned rather than guessed after the fact.
Systematic operator error also disappears: fatigue by the end of a shift and different interpretations of disputed cases no longer affect labeling.
The model responds the same way to the first and last frame of the stream, latency is fractions of a second per image, so the bottleneck becomes the transmission channel, not recognition.
What becomes measurable after launch:
- the share of frames going to automation without human involvement;
- reduction of manual review with the same stream volume;
- error rate on the control set broken down by class;
- response time per frame during streaming processing.
We select the architecture based on the nature of the data, not by default: ResNet and related families handle small objects, noise, and blur differently — we confirm the choice by comparison on the control set, not by a general recommendation.
Training approaches: custom network, fine-tuning, or ready-made service
The three options differ not in “quality” but in the amount of your labeling and how close the target classes are to what open models already can do.
Training from scratch makes sense when tens of thousands of labeled examples have been collected and the classes are specific: defects in images, rare textures, medical images.
If there is no such labeling, an attempt to grow a network from scratch results in overfitting and accuracy lower than that of a ready-made architecture.
Fine-tuning a model on a pretrained architecture covers most tasks: convolutional families are robust on small sets, transformer ones win when there are many classes and small details matter.
A ready-made recognition service is used for typical categories — where control over labeling and fine-tuning is not needed.
| Option | When it fits | Limitations |
|---|---|---|
| Training from scratch | custom labeling from tens of thousands of examples, classes unlike public sets | long cycle, need for data reserve and compute resources, high risk of overfitting |
| Fine-tuning a pretrained architecture | from several hundred labeled examples, classes close to natural images | labeling quality is critical, accuracy drops when cameras or lighting change |
| Ready-made recognition service | typical categories, need for a quick start without your own infrastructure | closed set of classes, no access to the model, dependence on the vendor |
We usually start with fine-tuning and measure accuracy on the holdout set: if it is not enough, we selectively expand labeling rather than changing the approach entirely.
How we train and implement the model: stages of work
The order of stages is subject to one constraint: the quality metric and the method of calculating it are fixed before the first training run.
If they are defined afterwards, any refinement turns into fitting to a specific sample — on streaming data accuracy drops, and the cause can no longer be explained.
- Data audit. We look at what actually exists: volume by class, labeling quality, duplicates, corrupted files, imbalance, and leakage between the training and validation parts.
- Quality metric. We take not “accuracy in general,” but a value tied to the cost of an error: the miss rate for a critical class, macro-F1 under class skew, the activation threshold.
- Baseline. A simple model or pretrained weights without fine-tuning — to know the bar from which improvement is measured and not confuse it with the effect of preprocessing.
- Training. We select the architecture for the task — convolutional (ResNet, EfficientNet) or transformer (ViT) — and the mode: first the top layers, then partial unfreezing, input normalization, class balancing, early stopping by validation.
- Holdout set. We conduct the final check on data that was not used in either training or parameter selection, and if possible — on images from a different period.
- Error analysis. We analyze the confusion matrix and the most confident mistakes: some are addressed by additional labeling and thresholding, others by changing the problem statement.
- Model implementation. Prediction service, weight versioning, confidence threshold and abstention instead of a random label, logging, input data drift monitoring, and a fine-tuning policy.
Training artifacts: model weights, labeling, retraining policy
After the project, the client is left not with “a trained model as a black box,” but with a kit that can be checked, fine-tuned, and handed over to another team. Below is what exactly is fixed and in what form.
- Model weights — trained parameters with a fixed list of classes and a confidence threshold below which the model honestly abstains instead of guessing.
- Training and launch code — readable scripts for data preparation, training, and prediction, without hidden external dependencies or manual edits “in place.”
- Labeled image set — training, validation, and test parts without intersections between them, so that metrics are not inflated by frame leakage.
- Labeling rules — how disputed frames, overlaps, and ambiguous classes are interpreted, which object is considered too small for labeling, and what to do with frames without an object.
- Quality report — confusion matrix by class: you can see where the model confuses neighboring categories and what data is missing for them.
- Retraining policy — triggers (a drop in the share of correct answers on the stream, data drift, the emergence of a new class), frequency of runs, and responsible people on the client side.
- Reproducible launch — fixed environment, random seed, and library versions: training is repeated with one command and gives a comparable result, not “roughly the same.”
- Documentation and handover — map of files and storage, procedure in case of failure, walkthrough of the solution with the client’s engineers.
Such a kit allows the model to be developed in-house or the contractor to be changed without losing prior work.
Case: product quality control on a production line
On the line, quality control relied on selective inspection: an operator removed every tenth part from the conveyor and assessed it visually.
A defect between checks moved further down the flow and surfaced at assembly — the batch had to be sorted through entirely, and by that time the cause of the failure was lost in statistics.
We break down a production case: how quality control was moved to image classification directly in the stream.
Labeling was built not from studio images, but from frames from the line — with glare on metal, conveyor vibration, and different lighting between shifts.
Borderline cases were labeled separately: scratches a fraction of a millimeter deep and acceptable machining marks that are not defects — without them, the classifier would trigger on normal parts.
The model was trained on the EfficientNet family, classes were balanced by frequency, and the decision threshold was selected by the “miss — false alarm” curve, not by a single accuracy value.
After launch, defect misses decreased by about six times, and false line stoppages dropped to two or three per shift instead of fifteen to twenty. Every item, not a sample, passes inspection — the share of inspected products increased from 10% to 100%.
The shift received not only the model: an interface with the verdict and highlighting of the problem area, an adjustable sensitivity threshold, and a log of triggers for reviewing disputed frames.
The update policy describes what to do when the raw material batch changes or lighting is repaired: the operator marks disputed frames, and every few weeks the classifier is fine-tuned on them. This keeps defect reduction from rolling back a month later.
How is fine-tuning different from training a model from scratch?
Fine-tuning is transfer of knowledge: we take a network that has already been trained on a large corpus of images and adapt it to our classes.
Training from scratch starts with random weights, and the network has to master everything at once: both basic features such as contours, textures, and gradients, and the specifics of your task.
When there is little data — hundreds or the first thousands of frames per class — fine-tuning almost always wins.
Early layers are frozen and left untouched, only the top layers and the classification layer are trained: the model converges in tens of epochs, and the risk of overfitting on a small sample remains manageable.
There is one condition — the classes must be similar to what the network has already seen: equipment, goods, documents, natural objects.
A custom network is needed when the subject area is far from ordinary photographs: thermal images, defects on metal, highly specialized imaging. Transfer of knowledge here carries the bias of the source corpus, and accuracy hits a ceiling.
The second condition is that there is enough data so that the model does not memorize the sample: usually this means tens of thousands of labeled frames plus noticeable class imbalance, which has to be balanced separately.
The practical criterion is the volume and homogeneity of data, accuracy requirements, and labeling cost. With a small dataset, we start with fine-tuning and only then gradually unfreeze layers if metrics stop growing.
Training from scratch without a sufficient corpus gives a worse result than carefully configured transfer of knowledge, so we compare both options on the holdout set and choose the one that holds up more robustly on the control data.
Addressing technical concerns: quality, data, support
Before the start, the client’s team asks the same engineering questions: is there enough labeled data, what to do with rare classes, how robustness is checked, and who fine-tunes the model after handover. Below are short answers on how this is solved in practice.
Will our data be enough for training?
We start from several hundred examples per class: we take pretrained weights and fine-tune the model on your set. We assemble validation stratified, and cover the lack of data with augmentations — while monitoring where overfitting begins.
What should be done with class imbalance?
We combine class weights in the loss function, resampling of rare examples, and selection of the decision threshold. We calculate quality by precision and recall, plus the confusion matrix for each class — otherwise a rare defect dissolves in overall accuracy.
How is model robustness checked?
We keep a separate set of images taken under other conditions: different lighting, angle, equipment. The gap between the training and validation sets shows overfitting before the model goes into operation and points to problematic classes.
Who fine-tunes the model after handover?
We hand over the weights, training configuration, dataset manifest, and launch script — the team fine-tunes on new data on its own. We support the first runs so that adding a class does not break the others.
Tell us about the task — we will come back with a plan and estimate
So that the project estimate is based on facts, not guesses, we need a minimum of inputs — usually one conversation and a couple of files are enough. These same data will form the basis of the technical specification, so we will collect them once and without repeated questions.
- Process description: where the image comes from, who and how makes a decision based on the model result.
- Sample images: several hundred frames from real conditions, including difficult ones — blurred, overexposed, with a partially occluded object.
- Current labeling: classes, dataset volume, who labeled and by what rule, what to do with disputed cases.
- Quality requirements: what error rate is acceptable and what is more expensive — missing an object or marking an extra one.
- Target environment: server, camera at the site, or mobile device, memory and response time constraints.
- Access and data: whether images can be processed on our infrastructure or must be kept within your perimeter.
- What has already been done: current code, previously trained models and their metrics — so as not to start from scratch.
Describe the task — we will come back with a work plan and estimate: stages, what we check at each, and what result we consider acceptance. If some data has not yet been collected, we will discuss the task and suggest what is critical and what can be clarified along the way.
