Time series anomaly detection: where monitoring breaks
Monitoring usually appears earlier than the model: dashboards exist, thresholds are set, yet an incident is found from a user complaint.
The reason is detection lag: the rule calculates an average over a 10-15 minute window and compares it with a static threshold, so smooth degradation gains strength long before it triggers. By the time of notification, losses have already accumulated.
The second failure point is false positives of monitoring. One threshold is applied to a series with daily and weekly seasonality, to regions with different load, and to a release period — night peaks and planned maintenance produce noise.
The on-call engineer receives dozens of alerts, only a small part is confirmed, and after a few weeks the signal stops being noticed.
Third — gap between data and alerts. Telemetry goes to one storage, rules live in the alerting system, and the mark about a node restart or configuration change does not reach analytics.
The series looks like an anomaly, although it is behind a known event, and feedback from the engineer to the model does not return — telemetry control remains manual.
The common thing in all three — the anomaly is assessed outside the context of the series, and time series anomaly detection degenerates into manual chart review.
We break down how this is closed: a baseline by seasonality, adaptive thresholds on a sliding window, labeling of confirmed and rejected triggers, and a single loop from telemetry ingestion to alerting.
Implementation effects: fewer false positives and downtime
The first effect is visible by false alarm rate. Instead of a hard threshold, the model predicts an expected corridor taking into account daily and weekly seasonality and trend, so an alert comes for deviation from the forecast, not for the growth of the value itself. Scheduled backup, report export, or a peak-hour spike stop waking the on-call engineer.
Second — time to deviation detection. Previously it surfaced through user reports or upon failure; now the median between the start of deviation and notification is measured in minutes.
A class of failures that is not caught by a threshold at all is addressed separately: slow drift of a metric, cumulative degradation, and desynchronization between related series. They do not cross an instantaneous boundary, but after hours they lead to a stop.
Third — load on the on-call shift.
We count alerts per shift and the share of those requiring real intervention: after tuning on historical windows, extra alerts go away, and the incident is handled at the degradation stage, when it is enough to switch a node or shed load, rather than restore data. Downtime measured in on-call person-hours is reduced precisely due to the early stage.
We verify this on retrospective data, not with words: models connect to a metrics storage such as Prometheus, run on labeled incidents, and before/after compare detection completeness, false alarm rate, and response time.
Batch, streaming, and hybrid detection modes: what to choose
Modes differ not by model power, but by the moment when the anomaly feature is calculated. Batch processing of a series recalculates the window entirely on a schedule — the model sees the whole context and confidently separates seasonality from an outlier.
Streaming detection evaluates each point on input: detection latency is measured in seconds, but the window is short, and a slow trend is not visible in it.
Hybrid mode separates tasks: a fast loop catches sharp outliers in the stream, while background recalculation over accumulated history refines thresholds and suppresses false positives.
It is more expensive to maintain — two loops, rule coordination, history storage — but only it covers cases where an anomaly manifests both instantly and as a shift over weeks.
| Mode | When it fits | Detection latency | Limitations and maintenance cost |
|---|---|---|---|
| Batch series processing | Retrospective analysis, reporting, incident investigation, daily and weekly seasonality | Minutes to hours, depends on recalculation schedule | Does not signal at the moment of failure; but there is only one loop — maintenance is simple |
| Streaming detection | Online monitoring where the moment itself matters: service response, sensor readings | Seconds from point arrival | Drift and gaps cause false positives; each stream node is a point of failure |
| Hybrid mode | Critical loops: both an instant signal and dynamics assessment are needed | Seconds on the fast loop, minutes to hours on the refining loop | Two loops, threshold coordination, history storage; more expensive to maintain |
The practical selection criterion is the acceptable time to signal and the cost of a false positive: it determines whether the second loop is justified.
Work sequence: from data audit to handover of procedures
The work sequence is the same for a pilot on one node and for deployment across the entire fleet — only the amount of history changes. Each stage ends with an artifact that is checked at the next step, so it cannot be skipped.
- Audit of historical data. We analyze archive depth, measurement frequency, gaps, and stuck sensors. Separately, we collect an incident log: what was considered a failure and who recorded it.
- Agreement on the quality criterion. We fix what is more expensive — missing an event or a false alarm. We measure quality by episodes, not by points: the detector must catch the event as a whole and not split it into a dozen triggers.
- Preparation of the reference sample. We label normal and anomalous segments, and put disputed cases separately. We set aside part of the history and do not touch it until the final check.
- Setup and calibration. We select sensitivity and thresholds for the agreed criterion. Seasonality, mode changes, and planned shutdowns must not read as a failure.
- Check on a holdout period. We run the detector on data it has not seen, including a season change. We look not only at accuracy, but also at the time margin before the event — whether the on-call engineer has time to react.
- Response procedure and handover. We describe who receives the signal, what to do on a trigger, and how a false alarm is marked. We hand over settings, documentation, and the procedure for updating on new data.
Handover package: configurations, diagrams, and procedures
A solution that cannot be handed over to another team remains under the control of its author only.
Therefore, together with the model, we hand over an engineering package: using it, the customer's on-call engineer will bring up the detector from scratch, change sensitivity, and investigate an incident without our support.
- Detector configurations — sensitivity thresholds, analysis windows, seasonality adjustment, noise suppression rules, and retraining schedule are stored in the repository as files, not in interface settings.
- Data flow diagrams — where series come from, where cleaning and aggregation take place, where anomaly scores go, and how the loop behaves when a source breaks.
- Access policies — roles for reading metrics and for changing thresholds are separated, connection keys are stored in a secure vault and rotated on a schedule.
- Solution documentation — architecture, feature set, experiment log with quality metrics, deployment procedure, and rollback to the previous version.
- Alerting procedure — criticality levels, signal routes, response time, deduplication rules, and the procedure for investigating false positives.
- Verification log — how to reproduce a past incident on a saved data snapshot and make sure that a threshold change has not broken neighboring metrics.
We review the package at acceptance together with the customer's team: the work is not considered delivered until their engineer brings up the detector and investigates a test incident independently.
Case study: eliminated nighttime false positives in the stream
Industrial loop: several hundred sensors — pressure, temperature, vibration — send telemetry to Prometheus, and the operator monitors charts on a shared dashboard.
Alerts were set on static thresholds selected manually: the boundary held until the first change in the line's operating mode.
What it was before: nighttime shifts blind
At night the line switches to another mode, and thresholds set during the day produced up to 7 false alarms out of 10. The operator went to the area, checked the node, and returned with nothing — several dozen empty checks accumulated per shift, and attention to signals dropped.
Real deviations were lost: slow vibration drift approached the threshold and did not cross it — over a quarter, three bearing excursions beyond the operating boundary were missed this way.
What was changed and what the result was
In the first month after implementation, the model ran in shadow mode alongside the old thresholds, and discrepancies were reviewed manually — this gave an honest baseline for comparison.
Then signals were decomposed into a seasonal component: the daily and weekly profile is subtracted, boundaries are recalculated on a sliding window, and an alert is raised only for a sustained deviation of several points in a row. The threshold was calibrated by the target number of alerts per shift, not by accuracy on history.
Nighttime false positives dropped from 7 out of 10 to 1 out of 10, response time — from half an hour to a few minutes, and there have been no missed deviations over eight months of observation.
Nighttime callouts of the on-call shift decreased to one or two every two weeks, while the old threshold rules remained enabled as a safety net.
How a statistical threshold differs from a trained model
A statistical threshold and a trained model differ not by tuning accuracy, but by the source of the normal boundary. In the first case, it is set by an engineer with a rule based on historical data; in the second, by a function reconstructed from many observations.
A threshold is predictable and reproducible; a model is sensitive to weak deviations that a rule does not see.
A threshold is calculated from the mean and spread of a metric, often after removing seasonality and trend.
If the process is stable and failures manifest as a sharp crossing of the boundary, this is sufficient: the rule can be explained in one line, and the on-call engineer investigates a false positive in a minute.
Seasonality and trend are the main source of errors: an interval calculated over the entire series gives a different number of triggers in different periods.
A trained model predicts the next point and compares it with the fact, or reconstructs the density of normal behavior — for example, Isolation Forest. It catches smooth shifts, combinations of features, and violations of the shape of the daily profile.
The price is history for training, a retraining procedure, and reduced interpretability: incident investigation requires a separate procedure.
The working compromise is an adaptive threshold: boundaries are calculated from residuals after decomposition and a sliding window, following seasonality and trend, but remaining a verifiable rule.
A model is connected where the cost of a miss is higher than the cost of a false positive, and is kept as a second loop.
The choice is confirmed by metrics: share of detected incidents, number of false positives per shift, time to detection. If the model loses to a simple rule on these numbers, we do not complicate the loop.
Frequently asked questions before starting: data, labeling, thresholds
Before starting, the same doubts usually come up: where to get labels for rare anomalies, what to do with changing seasonality, and how to integrate the detector into a working loop.
There are few labeled anomalies — will the model train at all?
Training is based on normal behavior: the model describes a normal series and estimates deviation from it.
Labels are needed for threshold calibration — a few dozen confirmed cases are enough; after that, the operator replenishes them by marking triggers directly in the interface.
Seasonality changes — won't false positives increase?
The baseline is recalculated on a sliding window, and seasonal profiles are built separately by hour and day of week, so a smooth load shift is handled. Holidays and sales are moved to an event calendar, and a change in the loop's operating mode — to a separate retraining point.
How can this be integrated into an existing loop without downtime?
We start with shadow mode: the detector calculates on the data stream but does not switch anything. Then comes recommendation mode, where the decision remains with the operator, and only after that — automatic actions. Access is issued to a service account with minimal rights, and rollback is done by removing a flag.
What about quality control after launch?
We display false positive rate, detection latency, and coverage of analyzed segments on the dashboard, with an alert for degradation next to them.
Once a week we review triggers with the operator: disputed cases go into labeling, and thresholds are recalculated from accumulated labels.
Let's discuss the task: what we will clarify and what we will send in response
To come back with an estimate rather than clarifying questions, on the call we go over three things: where time series come from and at what frequency, whether there are labels from past incidents, and what is considered an anomaly in your terminology — a single outlier, a level shift, or a deviation from the seasonal profile. This is enough to understand the class of the task and not design unnecessary layers.
Requirements gathering is based on data, not wishes: we look at the retrospective, estimate the share of gaps and noise, and check whether the incident can be reproduced on history.
After that, we put together a work plan and timeline estimate — with stages, acceptance points, and an explicit list of what is needed from your side.
Together with the estimate and plan, we send:
- loop diagram: series sources, arrival frequency, storage, where the anomaly signal goes;
- list of quality metrics on labeled history — share of missed incidents and false positive rate;
- stage-by-stage work plan: data preparation, baseline model, threshold tuning, pilot operation, handover;
- timeline estimate for each stage and a list of places where your decision is required;
- data requirements: retrospective depth, incident labeling, access to storages;
- operating mode options — periodic report or online signal with notification of the responsible person;
- rollback conditions and acceptance criteria by which you accept the result.
Describe the loop and data for time series anomaly detection — we will come back with an estimate and plan.
