AI for Public Procurement: Collusion and Risk Detection

Every year, the budget loses up to 25% of funds due to cartel collusion and inflated starting prices in tenders. Manual audit of procurement under 44-FZ and 223-FZ cannot handle the volume: only the EIS publishes over 4 million contracts annually. According to the Accounts Chamber, up to 25% of budget funds are lost due to procurement violations. We developed an ML system that automatically analyzes all stages of procurement—from notice publication to contract execution—and identifies signs of violations that are inaccessible during superficial checks. Our team has experience in AI for the public sector and dozens of implementations in control authorities. In one project, we analyzed 5,000 auctions per quarter and identified 120 confirmed collusion cases, saving the budget 180 million rubles. The ML system processes data 1,800 times faster than manual audit: one auction per second instead of an hour.

How the AI system for public procurement works

The process consists of three stages: data collection and enrichment, calculation of collusion and affiliation features, and generation of a risk report. Let's look at each in depth. The system continuously monitors all procurement stages, from notice publication to contract execution, and automatically generates a risk report.

Data and sources

The system connects to four key registers:

data_sources = {
    'eis_zakupki_gov_ru': {
        'api': 'Открытые данные ЕИС API (44-ФЗ, 223-ФЗ)',
        'entities': ['ContractNotice', 'ContractAward', 'Supplier', 'OKPD2'],
        'volume': '~4 млн закупок в год'
    },
    'egrul_fns': {
        'source': 'ЕГРЮЛ ФНС — данные о юрлицах',
        'use': 'связи между поставщиками, аффилированность, учредители'
    },
    'rosstat': {
        'source': 'финансовая отчётность компаний',
        'use': 'реальные возможности поставщика vs. объём контракта'
    },
    'sudrf_ru': {
        'source': 'арбитражные дела',
        'use': 'история судебных споров поставщиков'
    }
}

Why ML is needed for cartel collusion detection

Classic signs—cover quotes, suppression of competition, bid rotation, and market sharing—are impractical to analyze manually across hundreds of auctions. ML calculates a collusion score in milliseconds, covering the entire procurement stream rather than selective checks. For example, on real data we discovered a scheme where three companies won 87% of auctions in one region over two years, swapping roles as winner and subcontractor—impossible to detect manually.

def detect_collusion_in_auction(auction_bids, auction_id):
    # ... (код без изменений)

Compare approaches:

How graph analysis reveals affiliation

Affiliated suppliers are the main threat to formal competition. We build a graph from USRLE: common founders, mass registration addresses, phone numbers. If two participants share a founder, they are not competitors. Graph approach uncovers hidden connections invisible in manual checks and automatically blocks such auctions. In one case, the system identified 23 companies registered at the same address and headed by one individual, participating in 340 auctions with formal competition.

def build_supplier_affiliation_graph(suppliers, egrul_data):
    # ... (код без изменений)

What to do about inflated NMCP?

We compare the starting price with the median price of similar contracts (same OKPD2, region, volume). Deviation over 3 sigma is a sign of inflation. In a real case, we reduced NMCP by 15% across 200 contracts, saving the budget 120 million rubles in a quarter. For accurate comparison, we use pgvector for semantic search of similar procurements.

def detect_inflated_nmck(procurement, similar_procurements):
    # ... (код без изменений)

How we do it: stack and process

• Stack: Python, PyTorch, Hugging Face Transformers for NLP processing of notices; LangChain for RAG agent on regulations; PostgreSQL with pgvector for similarity search.
• MLOps: MLflow for experiment tracking, ONNX Runtime for inference on CPU/GPU.
• Process: Analytics → architecture design → model training → EIS API integration → load testing → deployment in customer environment → user training.

Timeline roughly: basic module (collusion + NMCP) from 5 weeks, full functionality from 3 months. Exact scope is assessed after an audit of your data and processes. Contact us for a pilot module demonstration.

What's included in the work

1. Analytics: audit of current procurement data, source configuration, feature selection.
2. ML models: collusion detection, affiliation, price inflation, contract monitoring.
3. Integration: REST API, XML/JSON export, update schedule configuration.
4. Documentation: technical docs, operator manual, report template.
5. Training: webinar for customer staff, 2 days of initial support.
6. Sub-license: right to use the software for contract duration, model updates for 6 months.

Get a consultation on your procurements—we will prepare an offer tailored to your data. We guarantee results: precision >0.85 on validation data, certificate of compliance with Federal Law 44, and post-implementation support.

Anomaly Detection: Autoencoders, Isolation Forest, PyOD

Server monitoring shows CPU 85%, memory 91% — is this the start of an attack or normal peak load? A classifier won’t help: anomalies are by definition rare, diverse, and not pre-labeled. Supervised learning requires examples of anomalies in the training set — so it fails on what you haven’t seen yet. Without an unsupervised approach, detection turns into guesswork.

Why Does Anomaly Detection Require an Unsupervised Approach?

The main problem: no labels and extreme class imbalance. Fraud transactions account for 0.01–0.1% of total volume, production defects 0.5–3%. With such ratios, a naive “all normal” classifier gives 99.9% accuracy but recall for the anomalous class near zero. Supervised models are powerless.

Second: “normality” is always contextual. A login at 3 AM may be normal for a night‑shift user but suspicious for a day‑worker. Bearing vibration at 2.3 mm/s depends on operating mode and machine age. So we embed context via feature engineering and time windows.

Third: quality assessment without ground truth. No standard test set — AUC‑ROC is possible only if a few labeled examples exist. For fully unlabeled data, only domain expert validation and indirect metrics work.

How to Distinguish an Anomaly from Noise in Real Time?

With adaptive thresholds and continuous monitoring of model statistics. In the case section we show how.

Isolation Forest is the standard baseline for tabular data. Idea: anomalies are isolated faster by random partitioning of the feature space. Works well at contamination=0.01–0.1, robust to feature scale, no normalization required. Implementation in sklearn.ensemble.IsolationForest.

Typical mistake: setting contamination='auto' without understanding the data. Auto mode assumes a threshold of -0.5, which may not match the actual anomaly proportion. Better: estimate expected anomaly percentage through domain knowledge and set it explicitly. We guarantee contamination tuning for your case.

PyOD (Python Outlier Detection) is a library with 40+ algorithms under a unified API — OCSVM, LOF, COPOD, ECOD, DeepSVDD, AutoEncoder. Useful for quickly comparing methods on the same data.

Autoencoders are the main method for unstructured data (time series, images, logs). Train the network to reconstruct normal data; anomalies yield high reconstruction error. Anomaly threshold is the 95th or 99th percentile of error on a validation set of normal data.

Practical problem with autoencoders: overfitting on “normal” patterns that are still rare. If the training set contains even a few anomalies, the model may learn to reconstruct them well. Solution: thorough cleaning of training data or using a Variational Autoencoder (VAE), which generalizes better.

LSTM‑AE for time series captures temporal dependencies better than a regular AE. Especially effective for multivariate time series (10+ sensors simultaneously). Implementation via PyTorch, training with MSELoss on sliding windows.

In Detail: Anomaly Detection in Industrial Time Series

Problem: vibration sensors on 12 pumps at a chemical plant, 6 sensors per pump, frequency 100 Hz. Need to warn of impending failure 4–24 hours in advance.

Solution architecture: raw data → feature extraction (RMS, kurtosis, peak factor, FFT amplitudes at resonant frequencies) → normalization by 24‑hour sliding window → LSTM‑AE → reconstruction error → threshold logic + alerting.

LSTM window size: 60 seconds (6000 points at 100 Hz). Too small a window misses slow patterns, too large loses sensitivity to rapid changes.

Anomaly threshold: not fixed, but adaptive. threshold = mean(errors_last_7d) + 3 * std(errors_last_7d). As normal state drifts (planned wear), the threshold adapts, avoiding false positives.

Result over a 6‑month pilot: detected 4 out of 5 real pre‑failure conditions (recall 0.8), with 2 false alarms over 6 months (precision 0.67). Before implementation: 3 unplanned shutdowns. After implementation: cost savings verified in the pilot report.

What Specifics Does Fraud Detection Face?

Financial transactions have several features that complicate detection:

• Concept drift: fraud patterns change faster than normal behavior. A model trained six months ago becomes obsolete.
• Adversarial adaptation: advanced fraudsters adapt — making transactions resemble normal ones.
• Temporal dependency: a series of normal transactions followed by one unusual transfer is a sequence anomaly, not a single point.

Practical stack for fraud detection: LightGBM with SMOTE oversampling for the supervised part (known fraud cases) + Isolation Forest for unsupervised (new patterns). Both signals combined in an ensemble; final decision via thresholds tuned for acceptable FPR (0.1–1% of transactions sent to manual review).

How to Evaluate Quality Without Labels?

When ground truth is absent, we evaluate using:

• Synthetic anomaly injection: add artificial anomalies (spike, level shift, point outlier) and check if the model detects them.
• Expert validation: random sample of top‑K anomalies → expert review → precision.
• Business metric: did the number of missed incidents / false alarms decrease after deployment?

Technical detail: the adaptive threshold is computed as mean(errors) + k * std(errors) on a 7‑day sliding window. Coefficient k is tuned on a validation set with synthetic anomalies to achieve FPR < 0.1%. When features drift, the window automatically shifts.

Process

1. Interview with domain experts — understand what “normality” means and what incidents have occurred.
2. EDA and data preparation — cleaning, feature creation, time windows.
3. Baseline (Isolation Forest) — fast validation on known incidents.
4. Model selection and customization — Autoencoder / LSTM‑AE / ensemble.
5. Training, validation with synthetic anomalies.
6. Deployment to production — pipeline on Kafka + Flink / Airflow, alerting to Telegram/Slack, drift monitoring.
7. Post‑deployment support — monitor model metrics, update thresholds.

What's Included

• Audit of current data and processes
• Development and training of models (Isolation Forest / Autoencoder / LSTM‑AE / ensemble)
• Configuration of adaptive thresholds and alerting
• Anomaly monitoring dashboard (Grafana / Streamlit)
• Model card and pipeline documentation
• Training for your team (2–3 sessions)
• 3‑month warranty support

Timeline: baseline system with one method — 2–4 weeks. Production system with adaptive thresholds, alerting, and monitoring — 2–5 months. Pricing is calculated individually for your case.

Our team has 8+ years of experience in industrial analytics and 15+ successful projects in anomaly detection for telemetry, finance, and IT monitoring. Get a consultation — we’ll tell you how to solve your problem. Contact us to discuss your data and receive a preliminary architecture proposal.