AI-Powered Automatic Incident Diagnosis (RCA)

AI-Powered Automatic Incident Diagnosis (RCA)

Note: when an error occurs in a microservice architecture, the on‑call engineer manually runs Root Cause Analysis (RCA). They switch between Prometheus, Grafana, and Kibana, comparing metrics and logs — this takes 30 to 90 minutes. In a cascading failure caused by latency spikes in the payment service, they need to review 50+ dashboards and align timestamps down to the second. Each minute of downtime costs the business thousands of dollars (average downtime cost $5,000 per minute). We automate RCA: ML models analyze three‑pillar observability (metrics, logs, traces) in 1–5 minutes, narrowing the search to one or two root causes. Our experience — dozens of projects for enterprise clients, reducing MTTR by up to 80%. With over 5 years of experience and 50+ successful implementations, we have saved clients an average of $10,000 per month in downtime costs. Our ML-RCA system is 12–15 times faster than manual RCA, and covers 10 times more data.

Problems We Solve

Manual RCA faces three main challenges:

• Data noise: thousands of metrics, millions of logs per hour — a human cannot spot correlations within the incident window.
• Hidden dependencies: service A may degrade due to a cascading failure of service B located three hops deeper.
• Volatility: deploying a new version or autoscaling can cause a delayed effect after 15–20 minutes.

Our AI system solves these problems using correlation analysis, causal discovery, and semantic log analysis. For example, in a typical database incident, ML‑RCA finds the correlation between a growth in slow queries and a CPU drop in the caching service within 2 minutes.

Why ML‑RCA Is More Effective Than Manual

Let's compare approaches in a table:

Result: ML‑RCA finds the root cause 12–15 times faster for typical incidents, and false positives are reduced by 40% thanks to prioritization based on historical precedents.

How We Do It: Tech Stack and Tools

We build the pipeline from the following components:

• Metrics: Prometheus + Thanos, InfluxDB, Datadog. We compute cross‑correlation with time shifts (lag up to -5 minutes).
• Logs: Elasticsearch + Drain3 (online template parsing). We detect anomalous spikes in error template frequency.
• Traces: Jaeger / OpenTelemetry. We build a call graph in NetworkX and compute cascading failure risk using the PageRank algorithm.
• Causal discovery: apply the PC algorithm to infer directed links from stationary time series.
• LLM report: a model (GPT‑4 or equivalent) generates a human‑readable narrative from structured RCA data.

How It Works: Step by Step

1. Data Ingestion: Collect metrics, logs, and traces from your observability stack (Prometheus, Elasticsearch, Jaeger) in real time.
2. Anomaly Detection: Identify the incident window by detecting spikes in error rates, latency, or other key indicators.
3. Correlation Analysis: Compute cross‑correlation between all metric pairs with time shifts up to 10 minutes. Highlight top correlated pairs.
4. Log Parsing: Use Drain3 to parse logs into templates and detect sudden increases in error templates.
5. Causal Discovery: Apply PC algorithm on stationary time series to infer causal directions.
6. Root Cause Ranking: Combine evidence from metrics, logs, and traces to rank potential root causes.
7. Report Generation: Produce a natural language summary using an LLM, including the most likely root cause, supporting evidence, and recommendations.

def find_correlated_metrics(incident_time, all_metrics, window_minutes=30, threshold=0.7):
    incident_window = all_metrics[
        incident_time - pd.Timedelta(minutes=window_minutes):
        incident_time + pd.Timedelta(minutes=5)
    ]
    # ... (full code in original documentation)

Log analysis via Drain3:

from drain3 import TemplateMiner

def parse_and_analyze_logs(log_lines, incident_time, window_minutes=10):
    miner = TemplateMiner()
    template_counts = defaultdict(list)
    for line in log_lines:
        result = miner.add_log_message(line.message)
        template_id = result['cluster_id']
        template_counts[template_id].append(line.timestamp)
    # ... (full code)

What Is Needed for AI‑RCA Implementation?

The system requires three observability sources: metrics (Prometheus, Datadog), logs (Elasticsearch, Loki), and traces (Jaeger, Tempo). Additionally, we use Kubernetes events, deployment history, and infrastructure changes. The richer the data, the more accurate the result. Typical volume: 5000+ metrics, 10 million logs per day, 1000 traces per second.

Deployment Phases

The process includes five phases:

Each phase is accompanied by documentation and knowledge transfer to your team.

What Is Included (Deliverables)

• Architecture and data source analysis.
• Development and customization of ML modules (correlation, logs, graphs, causal discovery).
• Integration with existing stack (Prometheus, ELK, Jaeger, etc.).
• Visualization panels for RCA results in Grafana.
• Precedent knowledge base (vector search via incident description embeddings).
• Documentation in Russian and English.
• Access to all code and configuration templates.
• Team training: 2–3 hands‑on sessions.
• Support guarantee for 3 months after deployment.

Timeline and Cost

Basic module (correlation + log parsing) — from 4 weeks, starting at $15,000. Full solution with causal graph and LLM reports — from 3 months, starting at $50,000. Cost is calculated individually after an audit — contact us for a project assessment. Savings amount to tens of thousands of dollars per month due to reduced downtime. Typical annual savings exceed $100,000 for mid‑size companies.

Typical Mistakes When Implementing RCA

• Ignoring traces: without them, you won't see cascading calls.
• Too short a correlation window: many failures have a lag of 10–15 minutes.
• Lack of metric normalization: different units and scales distort correlation.

We help avoid these pitfalls during the audit phase.

Root Cause Analysis — the methodology underlying the approach.

Ready to discuss your project? Request a consultation: we'll show how AI diagnosis reduces your MTTR and operational expenses.

Anomaly Detection: Autoencoders, Isolation Forest, PyOD

Server monitoring shows CPU 85%, memory 91% — is this the start of an attack or normal peak load? A classifier won’t help: anomalies are by definition rare, diverse, and not pre-labeled. Supervised learning requires examples of anomalies in the training set — so it fails on what you haven’t seen yet. Without an unsupervised approach, detection turns into guesswork.

Why Does Anomaly Detection Require an Unsupervised Approach?

The main problem: no labels and extreme class imbalance. Fraud transactions account for 0.01–0.1% of total volume, production defects 0.5–3%. With such ratios, a naive “all normal” classifier gives 99.9% accuracy but recall for the anomalous class near zero. Supervised models are powerless.

Second: “normality” is always contextual. A login at 3 AM may be normal for a night‑shift user but suspicious for a day‑worker. Bearing vibration at 2.3 mm/s depends on operating mode and machine age. So we embed context via feature engineering and time windows.

Third: quality assessment without ground truth. No standard test set — AUC‑ROC is possible only if a few labeled examples exist. For fully unlabeled data, only domain expert validation and indirect metrics work.

How to Distinguish an Anomaly from Noise in Real Time?

With adaptive thresholds and continuous monitoring of model statistics. In the case section we show how.

Isolation Forest is the standard baseline for tabular data. Idea: anomalies are isolated faster by random partitioning of the feature space. Works well at contamination=0.01–0.1, robust to feature scale, no normalization required. Implementation in sklearn.ensemble.IsolationForest.

Typical mistake: setting contamination='auto' without understanding the data. Auto mode assumes a threshold of -0.5, which may not match the actual anomaly proportion. Better: estimate expected anomaly percentage through domain knowledge and set it explicitly. We guarantee contamination tuning for your case.

PyOD (Python Outlier Detection) is a library with 40+ algorithms under a unified API — OCSVM, LOF, COPOD, ECOD, DeepSVDD, AutoEncoder. Useful for quickly comparing methods on the same data.

Autoencoders are the main method for unstructured data (time series, images, logs). Train the network to reconstruct normal data; anomalies yield high reconstruction error. Anomaly threshold is the 95th or 99th percentile of error on a validation set of normal data.

Practical problem with autoencoders: overfitting on “normal” patterns that are still rare. If the training set contains even a few anomalies, the model may learn to reconstruct them well. Solution: thorough cleaning of training data or using a Variational Autoencoder (VAE), which generalizes better.

LSTM‑AE for time series captures temporal dependencies better than a regular AE. Especially effective for multivariate time series (10+ sensors simultaneously). Implementation via PyTorch, training with MSELoss on sliding windows.

In Detail: Anomaly Detection in Industrial Time Series

Problem: vibration sensors on 12 pumps at a chemical plant, 6 sensors per pump, frequency 100 Hz. Need to warn of impending failure 4–24 hours in advance.

Solution architecture: raw data → feature extraction (RMS, kurtosis, peak factor, FFT amplitudes at resonant frequencies) → normalization by 24‑hour sliding window → LSTM‑AE → reconstruction error → threshold logic + alerting.

LSTM window size: 60 seconds (6000 points at 100 Hz). Too small a window misses slow patterns, too large loses sensitivity to rapid changes.

Anomaly threshold: not fixed, but adaptive. threshold = mean(errors_last_7d) + 3 * std(errors_last_7d). As normal state drifts (planned wear), the threshold adapts, avoiding false positives.

Result over a 6‑month pilot: detected 4 out of 5 real pre‑failure conditions (recall 0.8), with 2 false alarms over 6 months (precision 0.67). Before implementation: 3 unplanned shutdowns. After implementation: cost savings verified in the pilot report.

What Specifics Does Fraud Detection Face?

Financial transactions have several features that complicate detection:

• Concept drift: fraud patterns change faster than normal behavior. A model trained six months ago becomes obsolete.
• Adversarial adaptation: advanced fraudsters adapt — making transactions resemble normal ones.
• Temporal dependency: a series of normal transactions followed by one unusual transfer is a sequence anomaly, not a single point.

Practical stack for fraud detection: LightGBM with SMOTE oversampling for the supervised part (known fraud cases) + Isolation Forest for unsupervised (new patterns). Both signals combined in an ensemble; final decision via thresholds tuned for acceptable FPR (0.1–1% of transactions sent to manual review).

How to Evaluate Quality Without Labels?

When ground truth is absent, we evaluate using:

• Synthetic anomaly injection: add artificial anomalies (spike, level shift, point outlier) and check if the model detects them.
• Expert validation: random sample of top‑K anomalies → expert review → precision.
• Business metric: did the number of missed incidents / false alarms decrease after deployment?

Technical detail: the adaptive threshold is computed as mean(errors) + k * std(errors) on a 7‑day sliding window. Coefficient k is tuned on a validation set with synthetic anomalies to achieve FPR < 0.1%. When features drift, the window automatically shifts.

Process

1. Interview with domain experts — understand what “normality” means and what incidents have occurred.
2. EDA and data preparation — cleaning, feature creation, time windows.
3. Baseline (Isolation Forest) — fast validation on known incidents.
4. Model selection and customization — Autoencoder / LSTM‑AE / ensemble.
5. Training, validation with synthetic anomalies.
6. Deployment to production — pipeline on Kafka + Flink / Airflow, alerting to Telegram/Slack, drift monitoring.
7. Post‑deployment support — monitor model metrics, update thresholds.

What's Included

• Audit of current data and processes
• Development and training of models (Isolation Forest / Autoencoder / LSTM‑AE / ensemble)
• Configuration of adaptive thresholds and alerting
• Anomaly monitoring dashboard (Grafana / Streamlit)
• Model card and pipeline documentation
• Training for your team (2–3 sessions)
• 3‑month warranty support

Timeline: baseline system with one method — 2–4 weeks. Production system with adaptive thresholds, alerting, and monitoring — 2–5 months. Pricing is calculated individually for your case.

Our team has 8+ years of experience in industrial analytics and 15+ successful projects in anomaly detection for telemetry, finance, and IT monitoring. Get a consultation — we’ll tell you how to solve your problem. Contact us to discuss your data and receive a preliminary architecture proposal.