AI data quality control: automation and anomaly detection

Data quality control: where pipelines break

A classic failure looks like this: a nightly load fails midway, it is restarted from the beginning — and duplicate records appear in the fact table, because the record has no stable key and the insert is not idempotent.

A duplicate does not break the report immediately: it doubles revenue for some orders and surfaces only during reconciliation with accounting, when the decision has already been made.

Gaps in extracts behave even more quietly. The source returns data page by page, the pagination traversal breaks on a timeout, the job is marked successful — and the data mart for a specific day is missing part of the transactions.

The report shows a decline that did not happen, while the analyst looks for the cause in demand instead of the message queue.

The third point of failure is discrepancies in reference data: regional teams create customer codes and product groups according to their own rules, table joins lose rows, and the consolidated mart is assembled from incompatible parts.

Data errors accumulate for months and do not manifest until calculations start being made from them.

The cost of this state: decisions are made based on numbers that cannot be trusted, reconciliation turns into manual investigation, and trust in reporting falls faster than its accuracy grows.

Data quality control is not a one-time cleanup, but a layer of checks at every transition between systems. Below we break down how this is addressed.

Data anomalies: what early detection provides

An anomaly found after loading costs many times more: the corrupted batch has already spread to marts, reports, and models, and it has to be investigated manually — row by row and through logs.

Early detection moves the check to the point of ingestion, before the record enters the storage. It is at this stage that a batch is cheapest to stop and return to the source.

This works on two layers. Declarative rules enforce hard invariants: required fields, ranges, referential integrity, key uniqueness.

On top of them is a statistical profile of the stream: value distribution over a window, share of empty and rare categories, number of unique values, ingestion rate, divergence from the reference dataset.

We describe checks in the open-source framework Great Expectations — the rules live next to the loading code and are versioned with it.

Outlier detection catches what violates no rule but deviates from stream behavior: growth in the share of empties, collapse of unique values, distribution shift.

Stream monitoring is performed for each source separately, so a deviation is visible against its own baseline, not the overall average. The alert arrives with context — source, batch, metric, magnitude of deviation.

Measurable effect: 85–95% of quality incidents are caught before loading, and the volume of manual reconciliations is reduced by 3–5 times.

Some classes of failures disappear entirely — silent zeroing of a column after a source change, duplicates due to a repeated extract run, type substitution when the format changes. The problem is no longer discovered from a report during a review meeting.

Verification methods: rules, statistics, models

Checks in a data quality pipeline fall into three families: hard validation rules, statistical profiles, and trained models.

A rule answers the question "does the value comply with the contract," a profile answers "has stream behavior shifted," and a model answers "does this resemble what we have seen before." In practice, they are combined according to the cost of an error, rather than chosen once and for all.

Betting on a single method produces a predictable failure: rules do not cover semantic anomalies, and a model without labels goes into false positives on a new data segment.

That is why the set of checks is assembled in layers and thresholds are configured separately for each layer.

Comparison of approaches

Threshold values are not set "by eye": they are derived from the historical spread of the metric and revised when the source or segment changes.

A rule triggers instantly and explainably, a profile needs an observation window, and a model requires regular validation on fresh data — this is what determines which layer to put first.

How implementation of checks proceeds by stages

Implementation does not start with code, but with an inventory: as long as it is unknown where data comes from and who changes it, any check catches symptoms instead of the cause.

Below is the sequence of work and checkpoints where the customer sees the result before final acceptance.

1. Data source audit. We go through the supplying systems: owner, update frequency, formats, schema change history. The output is a source registry with notes on where the data already diverges from reality.
2. Risk map. For each field, we assess the cost of an error — distorted reporting, incorrect write-off, integration failure. We write checks not "for everything," but for what is more expensive to miss.
3. Design of the check framework. We define rule types: requiredness, range, referential integrity, discrepancies between systems. For each — a trigger threshold and response method: blocking, quarantine, notification.
4. Configuration of checks. We connect rules to the stream for writes and reads, set the schedule and logging so that every trigger is explainable and reproducible.
5. Run on historical data. We run the rules on accumulated datasets: we see the real defect rate, filter out false positives, and obtain a baseline for comparison after launch.
6. Result acceptance. We compare the "before and after" state using agreed metrics: how many defects are caught, what the false-positive rate is, and how quickly the signal arrives.
7. Handover. We provide configurations, rule descriptions, and instructions for the on-call shift — how to investigate a trigger and when to call an engineer.

Implementation artifacts: rules, reports, regulations

After delivery, what remains in the customer's environment is not a presentation but a working kit: rules, descriptions, diagrams, and instructions.

All of it is stored in the repository next to the code, versioned, and changed through the usual approval of edits, not through correspondence.

• Data quality policy — which checks are mandatory for each dataset, at what thresholds a violation is recorded, who approves an exception, and how a rule is retired from operation.
• Dataset description — source, owner, field composition, allowable value ranges, update frequency, and the reports that depend on this dataset.
• Check rule sets — declarative checks for completeness, uniqueness, referential integrity, and distributions, tied to a specific dataset and owner.
• Flow diagram — where a record passes a check, where the rejected part goes, at which points the flow stops, and what dependent consumers receive at that moment.
• Incident report — failure type, how many records were affected, time to detection and resolution, and which rule was added based on the investigation.
• On-call instructions — the procedure for responding to a trigger: what to check, how to isolate the area, when to involve the data owner, and how to record the result.
• Rule change log — what changed and why, from what date the version is in effect, and how new rules passed validation on historical data.

The kit remains with the customer along with access and edit history. A new rule is validated on past data before it is enabled, and removing a rule returns the flow to its previous behavior with a single edit.

Case: how failures in reporting marts were eliminated

The reporting marts were assembled from a dozen sources, and discrepancies in them were found not by a check but by a complaint: the figure for a region did not match the facts, analysts went into the source tables, and the cause was reached on the second or third day.

By that time, the report had already gone upstairs, and the business was making decisions based on incorrect data. Each such investigation took several hours for two engineers.

What was changed

Checks were built into the assembly loop itself: control of load completeness, partition freshness, schema changes, and distribution of key metrics at every layer — from raw data to the mart.

Thresholds are calculated not as a constant but from the seasonal profile of the metric, so an evening rise in orders does not raise a false alarm.

Orchestration remained in Airflow: the check result became a condition for launching the next step, rather than a separate report off to the side.

When triggered, the problematic partition goes into quarantine, publication of the mart is blocked, and the on-call engineer receives an alert indicating the source, layer, and metric.

Rollback is performed to the previous correct partition — reporting continues to live on validated data.

What became

• Time to failure detection: from several days to 15 minutes — at the start of loading, not after a complaint.
• Share of anomalies caught: about 94% versus roughly one third with selective manual reconciliations.
• Manual investigations: from 28 per month to 3–4, and those are for new types of failures that have not yet been described by a rule.
• There was no need to expand the team for the increased data volume: new sources are connected using the check template within a day.

What to do if checks are already partially configured

Partially configured checks are not "half done," but a separate task: to understand what of the existing setup works and what creates false confidence in the data.

We start with an inventory — which rules are declared, which fields they are tied to, how often they triggered, and who responded to them. Such a review usually removes 20–40% of the rules: they duplicate each other, reference renamed fields, or have not triggered for years.

Moving checks to the ingestion point is the most visible part of the work. When a rule triggers at the input, rather than after loading into marts, one source has to be fixed instead of ten reports that have already gone out to users.

We separately address the issue of owners: a rule without an accountable person triggers into the void. Together with the customer, we establish who is responsible for each dataset and what share of rejected records is considered normal.

Technical doubts and limitations of methods

Any anomaly detector is a trade-off between misses and false alarms, and it makes sense to discuss it on specific data, not in a demo.

Therefore, we start with measurements: what share of triggers is confirmed by a real defect, how the metric behaves during quiet and peak periods, and what happens when the source or field schema changes. Below are the objections most often raised by engineering teams during the approval stage.

How are seasonal spikes and planned peaks accounted for?

The metric is compared not with the average across the entire history, but with the same period of the previous cycle and a moving window. A holiday increase in orders, a sale, or a planned migration is not considered an anomaly.

For a new source, the first runs are performed in baseline-learning mode until sufficient statistics are accumulated.

What happens to sensitive data?

Checks are performed in the customer's environment: only aggregates and quality metrics go outside, without source records and personal fields.

Rules are described declaratively, for example in Great Expectations, and live in a private repository along with the rest of the code — access is restricted, and changes go through review.

Will loading have to be stopped during implementation?

No. Checks are connected in observation mode: they run in parallel with the main flow and initially only log the result. Blocking of loading is enabled later and selectively — for those rules where the cost of a miss is higher than the cost of a delay.

Rolling back a rule or the entire set is done with a single configuration edit, without rebuilding the flow.

Let's discuss the task and come back with an estimate and plan

To ensure that the estimate of the scope of work is based on facts rather than assumptions, and that the implementation plan for checks relies on real data, a minimum of inputs is needed.

The closer the materials sent are to the working environment, the more accurately we will specify the composition of checks and the order in which they are launched.

• List of sources: databases, extracts, APIs, files, event streams — indicating who is responsible for each and how often the data is updated.
• Examples of extracts over several periods — raw, with defects and gaps. A cleaned sample will not show which anomalies need to be caught first.
• Current rules: what is already considered an error, where checks currently live — in loading code, in reports, or manually — and what is unsatisfactory about them.
• Descriptions of fields and reference data: types, requiredness, allowable values, relationships between tables.
• Response time requirements: which discrepancies are critical immediately, and which can be investigated in a batch once a day.
• Known incidents: cases where corrupted data reached a report or a client — these show the priority of checks.
• Access restrictions: where read-only is permitted, which systems cannot be changed, and who approves edits.

This is enough for us to come back with an estimate of the scope of work and a staged implementation plan for checks. After that, data quality support is built on metrics: share of anomalies caught, time from discrepancy occurrence to signal, and number of false positives.

Describe the task in free form — we will clarify the rest with questions.