AI system for predicting code defects with AUC-ROC 0.85+

Code defect prediction: where quality breaks

A defect almost never looks like a failure at the moment it appears. A five-line change passes review, a green test suite, and release, then surfaces several iterations later — in a module that formally no one touched.

The cost of fixing a defect grows with each stage: in a branch it means dependency analysis and rerunning checks; after release it means rollback, state restoration, and root cause analysis.

The main losses occur not in the code but in decisions about it. Review looks at a change in isolation, static analysis catches style and typical patterns, tests confirm what was already covered.

The question “where is a defect likely right now” remains unanswered: usually these are areas where high coupling, frequent changes, and weak coverage converge.

Code defect prediction closes exactly this gap — a model with AUC-ROC above 0.85 ranks files and modules by error probability, and reviewers' attention goes where it pays off.

Late detection of errors hurts release predictability. A post-release regression is not a single incident but a shift in the entire pipeline: the team starts over-insuring, lengthens the run, adds manual acceptance, and slows delivery.

The longer a defect remains unnoticed, the wider the circle of changes that will have to be rechecked along with it.

Below we break down how such a system works: what signals it reads from repository history, how it integrates into review, and what it shows the team before risk becomes an incident.

What the model delivers: measurable effect on review and releases

The effect of the model is measured not by the feeling that “things got cleaner” but by three numbers: how much time it takes to analyze one change, what share of errors reaches the user, and how many person-hours code review consumes.

Before implementation, we capture these metrics on your repository for a comparable period; after, we compare them on the same set of rules and with the same team composition.

The speed of change review increases through prioritization. The model flags fragments where defect probability is higher, and the reviewer starts with them instead of reading the entire change.

On typical changes, review time is reduced by one and a half to two times; the effect holds as long as the trigger threshold is tuned to your code style and does not drown in false signals.

Early defect detection changes the class of failures. Hints arrive at the stage when the change is still in the branch: null dereferences, unhandled exceptions, errors in condition boundaries, forgotten handling branches.

Such a defect costs a one-line fix, while the same defect in release turns into an incident with root cause analysis and version rollback.

The effort spent on code review decreases, but not to zero — the goal is for the team's attention to go to logic and design decisions rather than mechanical omissions.

Some comments are formulated before discussion, so review begins with a ready list of questions rather than blind reading.

We fix the target quality threshold in advance — AUC-ROC no lower than 0.85 on your stack. If it does not hold on real data, we further train the model before measuring the effect: otherwise defect prediction turns into noise, and people simply stop opening the hints.

Implementation options: pipeline, review, manual run

There is one model, but three modes of using it — and the mode determines what the team does with the prediction.

Automatic control at build time turns the score into a blocking gate, review hints leave the decision to a human, and an offline run produces a branch report without affecting the overall pipeline.

General limitations

The model relies on repository history: on new modules and rare change types confidence is lower, while on mass refactoring the share of false positives grows.

We select the threshold not by an overall quality metric but by the false-positive rate on your data, so for the first month the mode runs in observation: we measure accuracy, look at confirmed and rejected triggers, and only then enable blocking.

Implementation stages: from repository audit to handover

The order of work here is strict, and it does not match the order of “installed the model — looked at the metric.” Defect prediction relies on change history: if labeling is assembled from random changes, noisy merges enter the training sample and quality hits a ceiling that no tuning can later lift. That is why we start with data and checks, and set thresholds last.

1. Repository audit — we measure history depth, the share of merges and automatic changes, linkage with the issue tracker, availability of branches for the required period. The output is a map of which files and authors have enough data, and where features will be insufficient.
2. Change history labeling — we fix the rule: a defect is a change that was later fixed by a separate change or an issue marked as an error. Auto-formatting, renames, and mass replacements are excluded, otherwise the model learns cosmetics.
3. Check schema — we assemble change and context features: size and spread of the change across files, density of past fixes, module freshness, test coverage. We cut validation by time, not randomly, otherwise the model peeks into the future.
4. Training and quality control — the model trains on past releases and is validated on later ones. The target benchmark is separating defective and clean changes at a level above 0.85 by area under the curve, with separate control of the miss rate.
5. Trigger threshold tuning — we choose two boundaries: the soft one gives a warning in review, the hard one blocks merge. We select thresholds by the number of false positives per hundred changes so that checks do not slow the team down.
6. Acceptance — a run on changes from recent weeks, comparison with manual review, and analysis of discrepancies. We look at which defects we catch, which we miss, and why.
7. Handover — access, a retraining procedure on new data, documentation on labeling and thresholds. The customer's team can change the rules without us.

What remains in the repository after implementation

Implementation ends not with a demonstration of a working model but with a set of artifacts in the customer's repository. Everything that affects system behavior is kept under version control next to the code — configurations, check schemas, quality reports.

Then the result is reproducible without us: a new team reads the documentation and continues from the same place.

• Check configuration — trigger threshold, path coverage, exclusions for generated code and dictionaries. Changed by a repository edit, without rebuilding the model.
• Integration schema — where the check is embedded: local run before sending changes, server-side check in a merge request, “warning only” mode during the trial period.
• Model quality report — metrics on a holdout sample (AUC-ROC, recall and precision at the working threshold), breakdown by defect types and components, list of known limitations.
• Set of confirmed triggers — history of accepted and rejected comments, on which the model is further trained inside the customer's perimeter, without exporting code outside.
• Documentation — description of data preparation and features, retraining and quality-check procedure, instructions for local launch.
• Support procedure — frequency of threshold review, procedure for analyzing false positives, drift control points, and a rollback plan to the previous model version.

Case study: how the share of defects before release decreased

The task was one: to make defects become known during review, not after release.

Before starting, we captured baseline metrics on the project history over several months — this gave a starting point rather than a feeling that “things got better.” We looked at four values: the share of defects found before release, detection recall on a holdout sample, the number of false positives per hundred checks, and team hours spent on manual review per sprint.

What produced the result

The main gain came not from the model itself but from features from repository history: frequency of changes to a file, change size, module coupling, authorship, and results of past checks.

The model is trained on labeling of real defects and configured so that the signal arrives in the Merge Request before approval — through GitLab and its checks in the build pipeline.

Separately, we tuned the threshold: reducing false positives from 34 to 8 per hundred checks removed the main reason such systems are switched off after a month.

Reviewers spend the freed hours on architectural discussions and complex changes rather than proofreading repetitive edits. The model is further trained on confirmed and rejected triggers, so detection recall does not degrade as the codebase grows.

How does a trained model differ from static rules?

Static rules are hand-written conditions: “file longer than N lines,” “cyclomatic complexity above threshold,” “code changed, tests untouched.” This is how, for example, SonarQube works: the same set of checks is applied to every change, and thresholds are set by an expert for the repository state at the time of configuration.

Rules do not change along with the project. A complexity threshold reasonable for a monolith produces a stream of false positives in a system split into modules; the team's style shifts, and a check that caught real defects starts triggering on refactorings. People stop reading the warnings — the signal disappears.

The model learns from the change history of a specific repository. The label is taken from the same place: a change after which a fixing commit soon followed in the same code area is considered defective.

Features are change size, number of affected files, frequency of module changes, and presence of tests next to the changed code.

The output is defect probability for each change and an adjustable threshold in one place. Before a release, the threshold is raised, and the top 5-10% of changes by risk go to review; in normal mode it is lowered and the view is broader. A set of rules has no such knob: a check either triggers or it does not.

The difference is visible over time: rules become outdated silently, the model is further trained on fresh history, and quality is measured on a holdout sample — AUC-ROC from 0.85 on the customer's own data.

A drop in the metric is noticeable earlier than when people stop reading warnings. One does not cancel the other: rules remain a fast filter and part of the features, while the model sets priorities.

Technical concerns: labeling quality and false positives

Labeling comes from different sources: static analysis rules, incident history, reviewer edits — and a noticeable share of examples contradicts itself.

One reviewer marks a finding as critical, another closes it silently, and the model learns from this discrepancy, reproducing it in predictions.

That is why, before training, we bring labeling to a single specification: class descriptions, disputed cases into a separate category, overlap of samples between labelers, and measurement of their agreement.

The second question is trust in the numbers. We calculate metrics on a holdout slice that did not participate in training, and separately record the false-positive rate per hundred findings, because it is noise in reports that determines whether they will be used.

The third is quality drift control: code style and review rules change, so we regularly compare the feature distribution and the share of confirmed findings, not just once at acceptance. We connect the stack and rules from SonarQube as a signal source, not as a ready-made answer.

What level of false positives is acceptable?

We set the threshold not “on average” but by the cost of an error in a specific process: a missed defect in a critical module and an extra flag in tests have different costs.

The benchmark we agree on at the start is the share of confirmed findings and a predictable number of false positives per hundred. If the report turns into noise, the team stops opening it, and the system dies before it pays for itself.

What happens when code style and labeling change?

We build in drift monitoring from day one: we track the distribution of input features and the share of findings that developers confirm.

When it deviates from the baseline, we launch retraining on a fresh slice and compare metrics with the previous version — rollback to the previous model remains available at any moment.

Let's discuss the task: what we need from you to start

To come back with a timeline estimate and work plan for your repository, we need the inputs below. Code defect prediction is built on how defects arose before — so we rely on your history, not on general rules from documentation.

• Repository snapshot or read access: change history for one to two years, branches, release tags. On a short history, the model cannot distinguish a dangerous change from a routine one.
• Incident log and issue tracker with the link “change — defect — fix.” This is the basis of labeling: without it there is nothing to train the system on.
• Build and test pipeline reports: which checks failed, where failures were more frequent, how long the build took.
• Your definition of a defect and critical error classes: exception handling, resource management, value boundaries, concurrent access.
• Current review process: how many changes per week, who reviews them, how much time passes before an error is detected.
• Data and access restrictions: what cannot be moved outside the perimeter, where it is permissible to train the model, who sees the reports.
• Acceptance criteria: on which metric we consider the result achieved and where we embed the hints — before commit or as a comment to a merge request.

With this set, we come back with a plan: labeling schema, feature composition, integration points, and acceptance protocol on a holdout sample. Describe the task and send access — we will provide timelines and scope of work.