AI system audit: where answer quality breaks
The most expensive degradation in answer quality is the one that is invisible in average metrics. Two similar phrasings of the same question get different answers, and the system confidently produces both.
The cause is almost always compound: the search index was updated, the model request template changed, a filtering rule was added — and behavior diverged from what was recorded at launch.
Rare scenarios fail first and remain unnoticed the longest. As long as an AI system audit relies on averaged accuracy across the stream, failures on long context, mixed languages, and ambiguous questions dissolve into the overall number.
AI solution validation should follow scenario labeling: rare ones separately, high-frequency ones separately, and those where no answer should appear at all separately.
Response time and the failure rate grow slowly, so they are overlooked. The index grows, context gets longer, repeated model calls are added — and the request starts hitting the timeout limit.
Some of these cases the system closes with a polite “failed to get a response,” and a model failure stops being distinguishable from an integration failure or a safety-rule block.
Request-processing bottlenecks are usually distributed across three layers: context preparation, model invocation, and post-processing and checks.
The audit records system behavior before changes, captures a reference set of requests with expected answers, and shows at which layer quality is lost — in the data, the template, or the rules. Next, we break down how this is addressed by process.
Measurable effect: answer quality, speed, resilience
We start an AI system audit with a baseline: a reference set of real requests with labeled answers, a load profile, and a list of known failures.
We run tests and labeling in LangSmith — the measurement history stays with the client, and every change is checked against it, not “by eye.”
Answer quality in percentages
We label several hundred real questions by class: correct answer, incomplete, fabrication, correct refusal. The share of correct ones is calculated across the entire sample, so after each change its contribution is visible.
Response time under load
We look at the 95th percentile, not the average: the tail of the distribution is what the user notices. The profile replicates the peak scenario, so it is clear where the queue forms and what actually removes the delay — cache, context reuse, or fewer model calls.
Failure resilience
We test external model unavailability, connection drops, and incomplete data: whether a meaningful answer remains or the system falls back to a blank screen. The audit result is not individual fixes but a closed failure class.
Infrastructure load
We count the number of model calls per user answer and the growth in consumption when traffic doubles. This shows where there are unnecessary calls and where the resource is actually needed.
Predictability after updates
Before every release, there is a regression run on the reference set: if the share of correct answers falls below the recorded threshold, the update does not ship. Rollback to previous versions of prompts and models remains available.
Audit formats: express check, full review, monitoring
Audit formats differ not in report volume but in sampling depth and which risks they are even capable of catching.
A prototype and a system under traffic require different things: at an early stage it is more important to check the robustness of the logic on a small set of scenarios, while in production it is about seeing degradation on real data. Below are three modes and their limitations.
An express check takes several days and covers key paths: a set of control requests, answer boundaries, and behavior on incomplete or contradictory data.
A full review adds load runs, sample labeling, and adversarial input testing — its conclusions are sufficient for a decision to move into production.
Regular quality monitoring is not a one-time effort but a loop: metrics are captured from the stream, deviations go into a report, and questionable answers go to manual review.
| Format | When it fits | Limitations |
|---|---|---|
| Express check | Prototype before a pilot, change of model or prompts, evaluation of a third-party solution | Small sample: rare failures and systemic distortions are not visible in it |
| Full review | Before production launch, after a major rework, in a regulated environment | Requires access to data and a test environment; the snapshot is current as of the audit date |
| Regular quality monitoring | System in production, request stream changes, errors are costly | Requires labeling and a metric owner; finds degradation but does not replace a one-time review |
The formats do not compete: an express check is appropriate on a prototype, a full review before a production launch, and regular monitoring after. Each next stage builds on the results of the previous one rather than starting from scratch.
How the work is structured: from problem definition to report handover
An audit begins not with runs but with agreeing on criteria: what share of errors is acceptable on critical scenarios, which answers are unacceptable under the security policy, and what we consider quality degradation.
Without recorded numbers, the report turns into a debate about tastes — so the first stage is always about agreements.
- Criteria and audit boundaries. We record the scenarios, the acceptable error threshold, and the classes of unacceptable answers. The output is an acceptance matrix by which the result is accepted.
- Collecting metrics on the current state. We capture response latency, the failure rate and its types, and the cost of processing one request. This is the baseline against which we compare everything that follows.
- Audit scheme. We describe which components we isolate, where we substitute external sources, and how we reproduce identical conditions between runs.
- Test request set. We assemble a corpus of real requests and edge cases: empty input, long context, conflicting instructions, attempts to bypass restrictions.
- Fixes. We adjust system instructions, document labeling, cutoff thresholds, and filtering rules. Every change comes with a run — the effect is visible, not assumed.
- Control run. The same set on the new version, compared with the baseline by metrics and failure classes; we analyze discrepancies until the picture matches.
- Handover of regulations. We hand over the audit scheme, the test request set, the configuration, and the decision log — with these, the system will be accepted next time, without us.
Each stage ends with a verifiable artifact: an acceptance matrix, a metrics report, a scheme, a request set. Therefore, the final report does not say “it got better,” but by how much and on which scenarios.
Document package: risk map, access policies, regulations
After an audit, the client’s team is left not with a report for the shelf but with a working document set: it can be used to hand the system over to another engineer, pass an external review, and repeat measurements months later.
The files live in the repository next to the code and are updated with the release, not in a separate email.
- Risk map — for each scenario: how the failure manifests, which signal we catch first, who is responsible, and what they do. Priorities are ranked by probability and cost of consequences, not alphabetically.
- Solution scheme description — which models and services are involved, where data comes from, where responsibility boundaries between components lie, and who owns each area.
- Access policies and roles — who sees requests and answers, who reads logs, who changes the configuration, who approves edits to the model instructions. Permissions are granted at the minimum necessary level.
- Test request set — reference requests with expected behavior: a correct answer, refusal when data is insufficient, no leakage of system instructions. The set covers the main dialog branches and is expanded after every incident.
- Quality audit regulation — run frequency, threshold values for accuracy and failures, who reviews the result, and how quickly they respond to a drop.
- Response regulation — the procedure for rolling back to the previous configuration, recording the cause, and updating the risk map and test set based on the review findings.
- Development plan — which scenarios we add next, by which metrics we will know it is time to expand the system, and what this will require from adjacent systems.
The package is designed so that the team can then run the system on its own: the documents describe not our presence but their processes.
Case study: answer quality grew from 61% to 94% in two months
Starting point
The client came with a working assistant for an internal knowledge base: 39% of answers did not pass acceptance — fabricated links, outdated wording, refusal where data was available. Questionable cases went to the first line, and employees rechecked every answer manually.
We started the review with a reference sample: 480 real dialogs labeled by three classes — correct, partially correct, fabrication. Without it, any change turns into a debate about tastes, so the first artifact was a set of reference pairs “question — expected answer.”
What changed
The bottleneck was found in retrieval: fragments were cut by fixed length and split tables with regulations, so the model completed the context itself.
We switched to splitting by document structure and reranking what was found — the share of cases where the needed fragment did not make it into the context fell from 27% to 4%.
The second loop was control: the answer is checked for grounding in the source and completeness, and questionable cases go to review.
We put quality evaluation on a regular run in Ragas — metrics are calculated on the same sample, so regression is visible immediately after a change, not after a complaint.
Result
In two months, answer quality increased from 61% to 94%, and the failure rate decreased from 11% to 1.8%. The share of requests the assistant resolved on its own rose from 48% to 79% — the load on the first line almost halved.
The audit protocol and reference pairs were handed over with the solution: the client runs them independently and sees discrepancies without us. Such a loop survives a model change — the metric remains a point of reference.
What to do if the solution is already in production and cannot be changed
A working system is fixed behavior, integrations, and accumulated data.
Therefore, the audit begins not with “let’s rewrite everything,” but with an inventory of the system: where the input is, where preprocessing is, where the model is, where post-processing and rules are, and which block contributes most to errors.
We take measurements on the live stream and on a reference sample that we record before work begins: without this, there is nothing to compare “before” and “after” against.
The sample is assembled from rare and questionable cases, not only typical ones — otherwise metrics grow while production quality stays flat.
Next, the system is broken down into blocks, each of which is versioned and replaced in parts. A new model version, a different feature set, and modified rule logic are tested on a copy of traffic while the previous path serves users. For each block we record a contract: input fields, output guarantees, and a degradation criterion.
Rollout proceeds in portions: first a shadow mode, where the request goes to both versions and discrepancies are analyzed manually; then an increasing share of traffic under a behavior switch.
The service does not stop, and rollback to the previous version triggers automatically based on the failure rate, response latency, and growth in manual operator edits.
In parallel, we address what prevents changing the system safely: access to models and data, logging of requests and versions, and reproducibility of runs on historical data.
The outcome is a plan of reversible steps with a measurable result: quality on the reference sample, the share of automatic answers, and the number of operator escalations.
Common technical concerns: privacy, labeling, data volume
Concerns at the start of an audit are usually engineering-related, not organizational: there is no reference labeling, data cannot leave the perimeter, and the current team is busy with the product. None of them blocks the work — each has an audit scheme that does not require stopping development.
How do you evaluate quality if no reference labeling exists?
We assemble it at the start: a stratified subsample of 300–500 examples, double-blind labeling, and calculation of inter-annotator agreement. With agreement above 0.8, the set becomes the reference, and accuracy, completeness, and answer calibration are calculated on it.
Then discrepancies are analyzed by error class — it becomes clear where the model errs systematically and where it is labeling noise.
How do you verify data privacy without exporting data outside?
The audit runs in your environment: our engineers work under separate read-only roles, all data access is recorded in an access log, and only aggregates and metrics without source records go outside.
If export is fundamentally necessary, anonymization is performed on your side according to pre-agreed rules.
Can an audit itself cause data leakage through requests?
We perform checks for training-example recovery and leakage through requests on synthetic and anonymized sets — they reproduce the data structure but not its content. Logs of test requests remain in your storage and do not leave the perimeter. Separately, we check whether the model returns fragments of input documents in its answers.
Will the sample size be sufficient, and will we get in the way of the current team?
We calculate the volume from the target metric error and the number of rare classes, not “the more, the better”: to assess quality and stability, a subsample covering all scenarios is usually sufficient.
Checks are launched in agreed load windows, data access is read-only, and the team’s release cycle is not shifted.
Let’s discuss the task: we will review the system and come back with a work plan
So that the timeline estimate is based on facts rather than assumptions, we start with diagnosing the AI solution based on what already exists in the project.
Nothing needs to be specially prepared: a code repository, configurations, and a description of how the system currently works are enough.
To discuss the task concretely, it would be useful for us to see:
- a description of the scenario: which requests come in and which answer is considered correct;
- how the system is structured — where model calls are, where rules are, where external services and storage are;
- logs of requests and answers for the period where behavior seems unstable;
- quality metrics, if they are already calculated: share of correct answers, failure rate, response time;
- access rules and data restrictions — what cannot leave the perimeter;
- current checks: automated tests, manual runs, analysis of user requests;
- timeline constraints and what changes most often in the system.
After the review, we come back with a work plan: stages, the order of checks, what we close at each step, and where decisions are needed on your side. The timeline estimate in it is tied to the size of the system and the criticality of the paths, not to average market numbers.
Describe the task in two paragraphs — we will come back with an estimate and a plan.
