Anomaly Detection in Time Series: Hybrid Detector

Imagine an infrastructure monitoring service generating hundreds of alerts daily, 90% of which are false. When metrics contain trends, seasonal spikes, and concept drift, static thresholds yield over 60% false alarms. In one project with 500 metrics, engineers spent 2 hours daily filtering alerts. After integrating a hybrid detector, triage time dropped to 15 minutes, and the false positive rate fell by 70%. The reduction in alert processing costs reaches 80%, with annual savings of up to $100,000 for a typical project. According to NIST research on time series, combining statistics and machine learning is the best approach for anomaly detection.

We are a team of AI engineers with 10+ years of experience in time series analysis and 50+ successful projects. We guarantee detection accuracy of at least 95% on your data. Contact us for a consultation and project assessment.

Types of Anomalies

Outlier detection starts with proper classification.

Point anomalies (outliers): A single value dramatically deviates from the series. Example: a temperature sensor reading 200°C when the norm is 50°C.

Contextual anomalies: A value is normal by itself but anomalous in context. Example: a temperature of 35°C in January (normal in summer, anomalous in winter).

Collective anomalies: A sequence of values is normal individually but anomalous together. Example: several standard transactions forming a fraud pattern.

Why STL + Isolation Forest is the Gold Standard

STL decomposition (Seasonal-Trend decomposition using Loess) splits the series into trend, seasonality, and residual. Anomalies are searched in the residual — this eliminates false positives from seasonal peaks. Isolation Forest on residuals effectively catches points that do not fit the normal distribution. For streaming data, we add an online Z-Score with an adaptive threshold.

This hybrid is faster than LSTM (milliseconds per point) and requires less data. In our projects, it achieves precision >0.95 and recall >0.9. STL + Isolation Forest is our primary choice for most tasks.

Detection Method Comparison

Typical Performance on Industrial Data

How to Choose a Detection Threshold Without Going Crazy

The threshold balances missing anomalies (False Negative) and false positives (False Positive). The optimal threshold depends on business goals: for critical metrics (service downtime), recall is more important; for sales monitoring, precision is key. We use a validation set and tune the threshold by F1-score or by precision at the N-th quantile. In production, the threshold adapts via a feedback loop: engineers label alerts, and the model retrains.

import numpy as np
from scipy.stats import median_abs_deviation

def zscore_anomalies(series, threshold=3.0):
    z_scores = np.abs((series - series.mean()) / series.std())
    return z_scores > threshold

def mad_anomalies(series, threshold=3.5):
    median = np.median(series)
    mad = median_abs_deviation(series)
    modified_z = 0.6745 * (series - median) / mad
    return np.abs(modified_z) > threshold

def cusum_detector(series, k=0.5, h=5.0):
    mean = series[:50].mean()
    std = series[:50].std()
    S_pos = np.zeros(len(series))
    S_neg = np.zeros(len(series))
    for t in range(1, len(series)):
        xi = (series[t] - mean) / std
        S_pos[t] = max(0, S_pos[t-1] + xi - k)
        S_neg[t] = max(0, S_neg[t-1] - xi - k)
    return (S_pos > h) | (S_neg > h)

from statsmodels.tsa.seasonal import STL

def stl_anomaly_detection(series, period=24, threshold=3.5):
    stl = STL(series, period=period, robust=True)
    result = stl.fit()
    residuals = result.resid
    mad = median_abs_deviation(residuals)
    modified_z = np.abs(0.6745 * (residuals - np.median(residuals)) / mad)
    return modified_z > threshold, result

from sklearn.ensemble import IsolationForest

def isolation_forest_detector(series, contamination=0.05, window=10):
    features = []
    for i in range(window, len(series)):
        window_data = series[i-window:i]
        features.append([
            window_data.mean(),
            window_data.std(),
            window_data.max() - window_data.min(),
            window_data[-1] - window_data.mean(),
            np.corrcoef(np.arange(window), window_data)[0,1]
        ])
    features = np.array(features)
    iso_forest = IsolationForest(contamination=contamination, random_state=42)
    predictions = iso_forest.fit_predict(features)
    return predictions == -1

import torch
import torch.nn as nn

class LSTMAutoencoder(nn.Module):
    def __init__(self, input_size, hidden_size=64, num_layers=2):
        super().__init__()
        self.encoder = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
        self.decoder = nn.LSTM(hidden_size, input_size, num_layers, batch_first=True)

    def forward(self, x):
        _, (h_n, c_n) = self.encoder(x)
        decoder_input = h_n[-1].unsqueeze(1).repeat(1, x.size(1), 1)
        reconstruction, _ = self.decoder(decoder_input)
        return reconstruction

def detect_autoencoder_anomalies(model, series, threshold_quantile=0.95):
    with torch.no_grad():
        reconstruction = model(series)
        re = torch.mean((series - reconstruction)**2, dim=[1, 2])
    threshold = torch.quantile(re, threshold_quantile)
    return re > threshold

Deliverables

Our deliverable package includes:

• Anomaly detector code for monitoring metrics (Python, deployment-ready)
• Dashboard in Grafana + alerting (Telegram, Slack)
• Documentation on thresholds and adaptation
• 2-hour training for your team
• Support for 2 weeks after deployment

Implementation Process: From Audit to Deployment

1. Analytics — collect historical data, identify anomaly types (point, contextual, collective), select metrics for monitoring.
2. Design — choose a combination of methods (STL, Isolation Forest, LSTM), define initial thresholds.
3. Development — write the detection pipeline, integrate with monitoring system (Prometheus, Grafana).
4. Testing — validate on historical data, A/B test in parallel mode, analyze false positive rate.
5. Deployment — install on staging, then production, configure alerts.
6. Monitoring — collect feedback, adapt thresholds, retrain models upon concept drift.

Timelines and Cost

Project costs start from $15,000 for the basic version (STL + Isolation Forest + dashboard) and from $50,000 for the full version with LSTM Autoencoder, streaming detection, and feedback loop. Order an anomaly detection system implementation today.

Anomaly Detection: Autoencoders, Isolation Forest, PyOD

Server monitoring shows CPU 85%, memory 91% — is this the start of an attack or normal peak load? A classifier won’t help: anomalies are by definition rare, diverse, and not pre-labeled. Supervised learning requires examples of anomalies in the training set — so it fails on what you haven’t seen yet. Without an unsupervised approach, detection turns into guesswork.

Why Does Anomaly Detection Require an Unsupervised Approach?

The main problem: no labels and extreme class imbalance. Fraud transactions account for 0.01–0.1% of total volume, production defects 0.5–3%. With such ratios, a naive “all normal” classifier gives 99.9% accuracy but recall for the anomalous class near zero. Supervised models are powerless.

Second: “normality” is always contextual. A login at 3 AM may be normal for a night‑shift user but suspicious for a day‑worker. Bearing vibration at 2.3 mm/s depends on operating mode and machine age. So we embed context via feature engineering and time windows.

Third: quality assessment without ground truth. No standard test set — AUC‑ROC is possible only if a few labeled examples exist. For fully unlabeled data, only domain expert validation and indirect metrics work.

How to Distinguish an Anomaly from Noise in Real Time?

With adaptive thresholds and continuous monitoring of model statistics. In the case section we show how.

Isolation Forest is the standard baseline for tabular data. Idea: anomalies are isolated faster by random partitioning of the feature space. Works well at contamination=0.01–0.1, robust to feature scale, no normalization required. Implementation in sklearn.ensemble.IsolationForest.

Typical mistake: setting contamination='auto' without understanding the data. Auto mode assumes a threshold of -0.5, which may not match the actual anomaly proportion. Better: estimate expected anomaly percentage through domain knowledge and set it explicitly. We guarantee contamination tuning for your case.

PyOD (Python Outlier Detection) is a library with 40+ algorithms under a unified API — OCSVM, LOF, COPOD, ECOD, DeepSVDD, AutoEncoder. Useful for quickly comparing methods on the same data.

Autoencoders are the main method for unstructured data (time series, images, logs). Train the network to reconstruct normal data; anomalies yield high reconstruction error. Anomaly threshold is the 95th or 99th percentile of error on a validation set of normal data.

Practical problem with autoencoders: overfitting on “normal” patterns that are still rare. If the training set contains even a few anomalies, the model may learn to reconstruct them well. Solution: thorough cleaning of training data or using a Variational Autoencoder (VAE), which generalizes better.

LSTM‑AE for time series captures temporal dependencies better than a regular AE. Especially effective for multivariate time series (10+ sensors simultaneously). Implementation via PyTorch, training with MSELoss on sliding windows.

In Detail: Anomaly Detection in Industrial Time Series

Problem: vibration sensors on 12 pumps at a chemical plant, 6 sensors per pump, frequency 100 Hz. Need to warn of impending failure 4–24 hours in advance.

Solution architecture: raw data → feature extraction (RMS, kurtosis, peak factor, FFT amplitudes at resonant frequencies) → normalization by 24‑hour sliding window → LSTM‑AE → reconstruction error → threshold logic + alerting.

LSTM window size: 60 seconds (6000 points at 100 Hz). Too small a window misses slow patterns, too large loses sensitivity to rapid changes.

Anomaly threshold: not fixed, but adaptive. threshold = mean(errors_last_7d) + 3 * std(errors_last_7d). As normal state drifts (planned wear), the threshold adapts, avoiding false positives.

Result over a 6‑month pilot: detected 4 out of 5 real pre‑failure conditions (recall 0.8), with 2 false alarms over 6 months (precision 0.67). Before implementation: 3 unplanned shutdowns. After implementation: cost savings verified in the pilot report.

What Specifics Does Fraud Detection Face?

Financial transactions have several features that complicate detection:

• Concept drift: fraud patterns change faster than normal behavior. A model trained six months ago becomes obsolete.
• Adversarial adaptation: advanced fraudsters adapt — making transactions resemble normal ones.
• Temporal dependency: a series of normal transactions followed by one unusual transfer is a sequence anomaly, not a single point.

Practical stack for fraud detection: LightGBM with SMOTE oversampling for the supervised part (known fraud cases) + Isolation Forest for unsupervised (new patterns). Both signals combined in an ensemble; final decision via thresholds tuned for acceptable FPR (0.1–1% of transactions sent to manual review).

How to Evaluate Quality Without Labels?

When ground truth is absent, we evaluate using:

• Synthetic anomaly injection: add artificial anomalies (spike, level shift, point outlier) and check if the model detects them.
• Expert validation: random sample of top‑K anomalies → expert review → precision.
• Business metric: did the number of missed incidents / false alarms decrease after deployment?

Technical detail: the adaptive threshold is computed as mean(errors) + k * std(errors) on a 7‑day sliding window. Coefficient k is tuned on a validation set with synthetic anomalies to achieve FPR < 0.1%. When features drift, the window automatically shifts.

Process

1. Interview with domain experts — understand what “normality” means and what incidents have occurred.
2. EDA and data preparation — cleaning, feature creation, time windows.
3. Baseline (Isolation Forest) — fast validation on known incidents.
4. Model selection and customization — Autoencoder / LSTM‑AE / ensemble.
5. Training, validation with synthetic anomalies.
6. Deployment to production — pipeline on Kafka + Flink / Airflow, alerting to Telegram/Slack, drift monitoring.
7. Post‑deployment support — monitor model metrics, update thresholds.

What's Included

• Audit of current data and processes
• Development and training of models (Isolation Forest / Autoencoder / LSTM‑AE / ensemble)
• Configuration of adaptive thresholds and alerting
• Anomaly monitoring dashboard (Grafana / Streamlit)
• Model card and pipeline documentation
• Training for your team (2–3 sessions)
• 3‑month warranty support

Timeline: baseline system with one method — 2–4 weeks. Production system with adaptive thresholds, alerting, and monitoring — 2–5 months. Pricing is calculated individually for your case.

Our team has 8+ years of experience in industrial analytics and 15+ successful projects in anomaly detection for telemetry, finance, and IT monitoring. Get a consultation — we’ll tell you how to solve your problem. Contact us to discuss your data and receive a preliminary architecture proposal.