Why training an anomaly detection model breaks down without labels
Manual defect labeling looks like a workable solution until you estimate the amount of work. An operator reviews hours of sensor recordings to find isolated deviations: the defect rate is fractions of a percent, sometimes one case per tens of thousands of cycles.
The output is many examples of normal behavior and only a handful of anomalies, on which a robust classifier cannot be built.
Class imbalance turns training into a trap. A model penalized for every error reduces loss fastest by declaring everything normal: formally high accuracy with zero usefulness.
Missed rare failures are invisible in average metrics—they are diluted by thousands of correct answers on normal data.
Shifting the threshold solves one problem and creates another. Lower sensitivity—detector false alarms stream in, the operator gets used to ignoring signals and misses a real failure amid the noise.
Raise it—missed rare failures increase, and their cost is an order of magnitude higher than a false alarm.
The labeling itself is also unstable: anomaly boundaries are fuzzy, two engineers label the same section differently, and some defect types do not occur at all in the collected data—there is nothing to show the model.
That is why anomaly detection without labels is built not on selecting labels but on describing normal behavior and finding deviations from it. Below we break down how this is addressed.
Impact of implementation: response time and a disappearing class of failures
Threshold rules catch only what is described in advance: temperature exceeding a limit, pressure going out of range. Training on unlabeled data removes this limitation—the model builds a profile of normal behavior and treats a sustained divergence from it as an anomaly.
PyOD addresses this task with a ready set of algorithms, so the first working pipeline can be assembled in a matter of days.
The time to detect a deviation drops from hours to minutes: the model evaluates the stream in real time rather than recalculating daily aggregates.
Manual labeling almost disappears—the operator marks isolated ambiguous cases instead of labeling the entire sample, and its share falls from full coverage to a few percent of the data volume.
A separate effect is a disappearing class of failures. Slow degradations that look normal from a single sensor are caught by the joint deviation of features: what used to be found only after a shutdown now arrives as an early signal.
Robustness to data drift is provided by recalculating the normal profile on a sliding window—seasonal and gradual shifts do not turn into a storm of alerts.
The operator's workload also drops due to grouping: related alerts are combined into a single incident with a significance score, so the on-call operator sees a few stories instead of hundreds of identical messages.
False alarms are reduced severalfold—the threshold is selected on historical data rather than set manually.
Monitoring modes: comparison of approaches and their limitations
The monitoring mode is chosen based on two quantities: how many confirmed anomaly examples are available and how stable the normal profile is.
If there are no labeled cases at all, unlabeled training remains—the model builds a representation of normal behavior and treats everything that does not fit it as an anomaly.
When several dozen confirmed incidents accumulate, partial labeling yields noticeably fewer false alarms at the same recall.
Below are three modes and where each runs into limitations. The sensitivity threshold is set separately in each: it determines the balance between a missed anomaly and a false alarm, and data requirements grow along with accuracy.
| Monitoring mode | When it fits | Where it runs into limitations |
|---|---|---|
| Unlabeled training | There are no incidents or only a few; the normal profile is stable and well described by an autoencoder | No labels—there is no way to verify quality; drift of the normal profile breaks the model, requiring regular retuning |
| Partial labeling | There are dozens of confirmed anomalies and analyzed false alarms | The anomaly class is skewed: rare failure types will not be learned by the model; labels need updating when behavior changes |
| Hybrid monitoring scheme | Manageability is needed and a clear reason for each alert | More expensive to maintain: two pipelines must be coordinated, and rules need an owner |
The hybrid is assembled as follows: an unlabeled model catches a broad class of deviations, and on top of it rules and confirmed examples filter out known noise. This removes the main pain of unlabeled training—the inability to prove that detection actually works.
Next we will look at how such a pipeline is validated on historical data and what happens during drift.
Implementation stages: from data audit to model handover
Unlabeled data changes the order of work: the model does not know in advance what to consider an anomaly—it derives the normal profile from the observations themselves.
Therefore a mistake in features or threshold surfaces during operation, when the on-call shift drowns in false alarms. We run the project in steps, each with a verifiable result.
- Data audit. We analyze what data arrives and at what frequency, where there are gaps and communication breaks, and how the system behaves by day of week and season. We separately study the log of past incidents—it provides reference points, although it does not serve as labels.
- Solution design and feature selection. We determine which signals describe the normal profile, over which windows we aggregate, what we normalize, and what we discard as noise. We agree on the feature set with your team before training.
- Training on the normal profile. The model builds a description of typical behavior and treats everything that does not fit it as an anomaly. No labels are needed—a period of stable operation is sufficient.
- Threshold tuning. We select the alert threshold based on the cost of error: a missed incident versus an unnecessary shift callout. We fix the threshold numerically so that it is changed deliberately.
- Validation on historical data. We run the model over past periods and compare alerts with known events. We calculate the share of incidents found and the number of false alarms per shift.
- Pilot operation. We launch in monitoring mode, review every alert together with the on-call shift, and refine features and thresholds based on actual results.
- Model handover. We deliver the configuration, retraining procedure, quality metrics, and instructions for the on-call shift.
Deliverables: configurations, diagrams, procedures
We hand over the result not as a set of scripts but as a working system that your team can maintain independently.
Each artifact is tied to a specific action: restart training, explain to a new engineer what is considered normal, change a feature without rebuilding the entire pipeline. Below is what physically remains in your repository after handover.
- Detector configuration — a training profile with sensitivity thresholds, observation window, and score aggregation rules; parameters are divided into "touch during retraining" and "change only deliberately."
- Data flow diagram — from the event source to score recording: where cleaning happens, where the feature representation is calculated, and where training sample versions are stored.
- Feature description — a table "feature → source → calculation logic → behavior on missing values," with a note on which fields are critical and which can be excluded without loss of quality.
- Retraining procedure — triggers (share of rejected alerts, input data drift, scheduled update), the procedure for rolling out a new version, and the criterion for rolling back to the previous one.
- Alert review procedure — how a verdict is recorded for each anomaly and how that labeling is returned to the next training sample.
- Team instructions — starting training, checking quality on a holdout sample, reading logs, typical failures, and what to do for each.
- Monitoring diagram — which quality metrics we review and at what interval, and at what values we raise an alert.
- Decision log — why the current detector architecture was chosen and which alternatives were rejected, so that six months later the analysis does not start from scratch.
The package is designed for an engineer who has not worked with this system before: they can deploy the pipeline according to the instructions and perform the first retraining without contacting us.
Beyond that, it is a matter of your priorities: some teams maintain the model themselves, while others leave support and incremental training on new data to us.
Case study: equipment failures in continuous production
Continuous production: the line runs without stops, and equipment failure almost always begins with a small deviation—rising bearing vibration, creeping drive current, temperature drift.
Telemetry from drives and sensors (historians on Siemens controllers) had accumulated for years, but there were no labels saying "failure occurred here"—only repair logs with shift notes.
Before implementation, monitoring relied on thresholds for individual signals set "with a margin." There were 25–30 false alarms per shift, and after a week the operator stops responding to them—while 6 out of 10 early failures went unnoticed and progressed to an emergency line shutdown.
How the anomaly detection model was trained without labels
We took only periods of operation in normal mode and built a normal profile taking into account shift, load, and seasonal fluctuations: deviation is calculated from the combination of signals, not from a single sensor.
The threshold was set on a held-out period with known repairs, and we separately verified that the model does not trigger on scheduled mode transitions.
Metrics before and after on the same section:
- missed failures: 6 out of 10 before — 1 out of 10 after;
- false alarms: 25–30 per shift — 3–4, and the operator reads each one again;
- time from first deviation to intervention: about 40 minutes — 6–8 minutes;
- share of failures caught before component damage: 15% — 70%.
The signal set and threshold are rebuilt based on accumulated repairs: the longer the system runs, the more accurate the normal profile.
How does the unlabeled approach differ from conventional classification?
Classification by examples learns from "feature — label" pairs and memorizes the class boundaries it was shown.
For accidents and defects this works poorly: rare events are not fully described by any class, and the model confidently labels only what it has already encountered.
Anomaly detection is built in reverse. The model describes the normal profile—the typical behavior of a process or equipment—and marks as an anomaly everything that does not fit this description.
There is no need to label failures: accumulated "healthy" telemetry is sufficient, and there is usually an order of magnitude more of it.
| Option | When it fits | Limitations |
|---|---|---|
| Classification by examples | Both classes are frequent and similar to each other | Does not see a new failure type, overfits to the rare class |
| Unlabeled anomaly detection | Failures are rare, the normal profile is stable and accumulated | More false alarms, threshold tuning is needed |
| Labeling part of the alerts | An expert has time to confirm the sample | Labels are biased toward frequent failures |
| Rules and thresholds | Explainability and a baseline are needed | Breaks when the mode changes, rules quickly become outdated |
The choice of approach comes down to two questions: do you have a description of the normal profile, and how many labeled failures are available.
Ready-made implementations of the methods are collected in open libraries, such as PyOD, so comparing methods on your data takes hours, not weeks.
If labeling is incomplete, a workable scheme is first anomaly detection on unlabeled data, then incremental training on alerts confirmed by an expert. Comparison of methods then uses a single indicator: how many real failures are caught at an acceptable false alarm rate.
Technical concerns: false alarms and data drift
False alarms and data drift are two concerns that arise before launching anomaly detection on an unlabeled stream.
A metric on a holdout sample says little here: it does not show how many alerts the on-call operator will have to review or how the model will behave after a change in operating mode.
Therefore we make the launch decision based on the model's behavior on live data, not on a single number in a report.
The model will produce a flood of false alarms—how is this controlled?
We tune the alert threshold against the stream: we look at how many signals per thousand events each setting produces, and keep the one where the operator reviews the alert rather than getting used to ignoring it.
Events with a borderline score go into an ambiguity zone—for human confirmation—and the human's decisions are returned to training as labels.
What should be done when data changes over time?
We compare the feature distribution on a sliding window with a reference period and set up a divergence notification—it triggers before quality starts to decline in the customer's eyes.
We trigger retraining based on this trigger, not on a calendar: during stable months we do not touch the model, and when the mode changes we perform incremental training on accumulated feedback.
How much historical data is needed at the start?
Anomaly labels are not required—a clean background of normal operation is needed, covering all standard modes: weekdays and weekends, seasonal peaks, scheduled maintenance. If there are signed-off incidents in the history, we use them for validation, not for training. We fill in missing modes during the first weeks of observation.
Can the solution be deployed on our equipment?
Yes, we deploy the model and the entire processing pipeline within your perimeter: data does not leave your infrastructure, and your team retains access to it. We calculate resource requirements based on stream volume and allow headroom for peak periods. Model updates follow the procedure, with the ability to roll back to the previous working version.
Let's discuss the task and come back with an estimate and plan
To start the discussion, you do not need a prepared dataset or labeled corpus—what is already being collected in the system is enough.
Send an export of telemetry or logs for a typical period and an anomalous period, a list of objects and features you consider significant, and one or two descriptions of cases the detector must catch.
Labels are not needed for unsupervised learning: rare and "strange" sections of history are the training signal.
- Starting data: raw records over several weeks of observation, including outliers, downtime, and seasonal fluctuations, not just "clean" days.
- Process description: where events are recorded, which features are available continuously, and which arrive with delay or gaps.
- Success criterion: what is considered an anomaly, what share of misses is acceptable, and who on the customer side reviews alerts.
- Operating environment: where the detector will run—in your perimeter or ours—and what restrictions apply to access and history retention depth.
- Scope of work: which stages you are ready to handle in-house and which you hand over to us—data collection, tuning, support.
In response, we send a solution design, a stage-by-stage scope of work with acceptance points, and an estimate and work plan. Describe the task in free form—we will come back with an estimate and plan.
