Evaluating the quality of a fine-tuned model: why metrics lie
The held-out set and live traffic almost always diverge. On a fixed set, the model answers according to the template that we ourselves built into it during labeling, while production receives short queries without context, typos, mixed languages, and multi-turn dialogues.
A metric on a held-out set shows how well the model replicated the labeler, not how useful it is to the user.
After fine-tuning, the divergence increases. The model overfits to the answer format from the training examples and loses some basic capabilities: quality drops on rare scenarios, refusals and generation cutoffs increase, and adherence to the system instruction breaks.
Yet the report is all green — same set, same template — while the consequence of an error is a rolled-back release and lost trust in reporting.
We look at quality by layers: automatic metrics on a held-out set, LLM-as-judge evaluation with labeled reference answers, and manual expert review on disputed examples.
Additionally, we run the model in shadow mode on real traffic and compare it with the current release by segments — by scenario, dialogue length, refusal rate, and cutoff rate.
Model release acceptance is built as a set of thresholds, not as a single number. A release does not pass if degradation after fine-tuning affects protected scenarios, even when the average metric has increased.
This approach is described in open metric suites such as Ragas and is supplemented with our task-specific reference data.
What changes when evaluation runs on every release
Pre-release validation time — days instead of weeks
When the set of checks and metrics is fixed, it is not rebuilt for every release. The run follows the same scenario on the same data, and divergence from the previous version is visible in the report immediately.
Pre-release validation time is reduced from weeks to one or two days — equally for a fine-tuned model and for edits to its instructions.
The share of false releases drops
A metric on a held-out set, evaluation by a judge model, and blind expert review provide three independent signals.
If at least one does not pass the threshold, the release does not go to users — the decision becomes numerical rather than "it seems to have gotten better." Rollbacks after deployment become many times fewer: regressions are caught before release, not from support complaints.
Unified acceptance rules instead of verbal agreements
The acceptance threshold is fixed in advance: which metric, on which sample, with what tolerance. Unified acceptance rules remove dependence on who is checking today and how optimistic their report is. A debate about taste is replaced by reviewing deviations by the numbers.
Regular release evaluation accumulates history
Each run leaves a snapshot: model version, data composition, metric values. After several releases, it is clear which edits actually move quality and which are noise. The next evaluation cycle is faster because we check what has already shown an effect.
Options for release validation: automated tests, reference set, experts
Three validation methods answer different questions and do not replace one another. Automatic metrics catch regressions where the answer can be checked formally, but are blind to semantic shifts when the form is correct.
The judge loop covers free-form answers, but inherits the biases of the same model. Expert labeling provides a reference, but its resource is limited and does not scale to every run.
| Approach | When it fits | Limitations |
|---|---|---|
| Automatic metrics (exact match, format, F1) | Field extraction, classification, tool calls, answer schema validation | Does not see semantic errors when the form is correct; on open answers, almost uninformative |
| Judge loop on a language model | Free-form answers: tone, completeness, instruction following; running on hundreds of cases | Inherits the model's biases and blind spots; requires calibration against expert evaluations |
| Expert labeling | Reference for disputed cases, judge calibration, high-stakes answers | Slow, depends on agreement between experts, hard to cover the entire volume |
| Combination of layers | Release control: lightweight layer for every change, heavyweight — at boundaries | Need reference storage, run infrastructure, and discrepancy review |
How the layers combine
We separate them by run frequency: formal checks run on every change and cut off obvious breakages; the judge loop runs on an extended sample before release, where it is important to see semantic degradation; experts are involved on borderline cases and for judge calibration. This order provides fast feedback while relying on a human reference where it is truly needed.
From audit to handover: how the validation loop is built
The validation loop is built so that the acceptance decision relies on repeatable measurement, not on the impression from a successful demo.
The order of work is the same for any task — metrics, rubrics, and thresholds change, but the sequence remains: first we describe how quality is evaluated now, then we fix the reference, and only then do we automate.
- Process audit. We examine how quality decisions are made today: where evaluation is done by eye, where it is based on a single shown example, which defects are not recorded at all. The output is a map of places where the verdict is not reproducible.
- Reference set. We assemble a sample covering frequent scenarios, rare and borderline cases. The set is versioned together with the model — without it, comparing releases with one another is impossible.
- Labeling and expert agreement. Two or three domain specialists label examples independently, disputed cases are reviewed separately, and the agreement rate is calculated before work starts. If experts do not agree with one another, automatic evaluation will not provide a foundation.
- Automated validation. We add deterministic rules and validation by a second model: the judge receives the reference answer, rubric, and format requirements. The judge is calibrated on expert labeling, and if divergence exceeds a set level, the rubric is rewritten rather than the result being adjusted.
- Threshold calibration. For each error class, we set a limit separately: missing a critical defect is not compensated by an increase in the average score. Thresholds are fixed before the run.
- Regression run. The fine-tuned and baseline models are run on the same set, and the difference is calculated for each metric separately. Only what is confirmed on the reference counts as an improvement.
- Acceptance policy and handover. We deliver a document with the set composition, rubrics, thresholds, and the procedure for revising them, as well as rights to the runs.
After handover, the client runs validation themselves and sees deviation before deployment. The reference size and thresholds are revised as new scenarios appear.
Metric suite, reference data, and acceptance policy
We deliver not only the fine-tuned model but also its evaluation loop: after handover, the client's team can run a new version on their own data and get the same verdict as we do.
All definitions, configurations, and access rules are recorded in documents that do not need to be reconstructed from correspondence — it is enough to open and follow them.
- Metric registry — a table where, for each metric, the definition, calculation method, data slice, acceptance threshold, and owner are specified. Differences in interpretation are resolved before evaluation starts, not after a disputed release.
- Labeled reference set — a stratified held-out sample: rare classes and borderline cases are represented explicitly, each example is labeled by two experts, disputed ones are reviewed by an arbitrator, and labeler agreement is recorded as a number.
- Automated judge calibration report — how closely the machine evaluation matches expert evaluation: agreement by class, divergence rate, systematic bias in favor of long or confident answers.
- Data access policy — who sees the reference and on what basis, what is anonymized, how the access log is maintained. Separately, we close the risk of training on the validation set.
- Acceptance policy — which metrics and at which thresholds must align, what we do if one of them fails, and in which cases the version is rejected entirely.
- Loop documentation — run diagram, metric dictionary, labeler instructions, and launch checklist: the evaluation is reproduced by an engineer who did not participate in development.
- Configurations and evaluation launch — a single run on an anonymized slice produces a report in a fixed format, so the results of two versions are compared line by line.
Case study: release metrics before and after implementation
The client is a support service with a knowledge base of several thousand articles, on top of which a fine-tuned model operates. It answers not "in general," but according to specific documents, and each release is a new fine-tuning branch.
Quality had to be evaluated selectively and only after deployment: there was no reproducible set of checks.
Before implementation, an expert manually reviewed about two hundred dialogues per release — that is two working days and less than five percent coverage.
The rest was checked by feel, disputed answers were evaluated "by eye," and four out of ten releases were returned for rework after launch, when users saw the error.
How the measurements were built
We assembled a reference set of six hundred labeled dialogues: question, correct answer from the knowledge base, and what must not be in the answer. On it, we calculate three things — hitting the fact from the document, absence of invented details, and answer completeness.
The run was automated via Ragas, and initial labeling of disputed cases was given to LLM-as-judge: the expert reviews not the entire release, but only divergences from the model's evaluation, of which about ten percent remain.
What changed: release validation takes three to four hours instead of two days, coverage is one hundred percent of dialogues from the reference set.
Returns for rework decreased from four releases to one out of ten, and regression on past scenarios is visible even before deployment. The set is versioned together with the model, so two fine-tuning branches are compared on the same examples.
How does online monitoring differ from validation on a reference set?
A reference set is a fixed slice of labeled queries that is run before deployment. It answers the question "did we break what already worked?" and allows comparing model versions with one another on unchanged data.
Online monitoring answers a different question: after release, both user behavior and input data are no longer the same as they were during fine-tuning preparation.
| Loop | What it answers | What it does not catch |
|---|---|---|
| Reference set | Regression: did what worked break | Input data shift after deployment |
| Online monitoring | Drift: query distribution, refusals, answer format | Causality — there is a signal, but the cause is unclear |
Online monitoring calculates the refusal rate, answer completeness and length, calls to external sources, and the distribution of judge scores. When a metric leaves the specified range, drift tracking turns on, and disputed answers go to manual sample review.
Then the loops connect: reviewed cases after expert labeling replenish the reference set, so it grows together with the product instead of becoming outdated.
The first line of evaluation is held by LLM-as-judge on pairwise version comparison, the second by experts on a sample; metric calculation and run storage are covered by Ragas.
If only one loop is left, two predictable failures occur: the set imperceptibly turns into a museum of the past product, and the online signal turns into a stream of complaints that cannot be tied to anything.
Common technical questions about the validation loop
We assemble the validation loop before starting fine-tuning — otherwise metrics are unwittingly adjusted to the result already obtained.
During alignment, the same questions usually come up: reference set size, labeling without domain experts, noise in the sample, and transferring metrics between iterations.
What reference set size do we consider sufficient?
A guideline is 30-50 examples per each significant scenario; rare classes are collected separately, otherwise their metric fluctuates from iteration to iteration. The set is versioned and fixed with a checksum so that comparison of runs remains correct.
Can a sample be labeled without domain experts?
For formal criteria, yes: written instructions, two labelers, and a third vote in case of disagreement.
Domain requirements cannot be covered this way — a small expert calibration sets the reference, then the judge model scales up the volume under agreement control with the expert.
What to do about noise in the sample?
We stratify the set by scenario and remove nearly identical examples — they skew the success rate. Borderline cases are moved to a separate layer: an averaged metric without dispersion by subgroup does not show where the model breaks.
Can metrics from other projects be reused?
The framework transfers: checks for format, answer completeness, and refusals do not depend on the domain. We rebuild the reference examples, criteria, and trigger threshold for the specific task.
How does the client participate in validation?
The product owner approves the criteria and accepts the result on a closed reference set. Several hours of domain expert time are needed for calibration — after that, runs proceed without their involvement.
Let's discuss the task: what to send so we can return with a plan
To discuss the task substantively and return with a work plan rather than general considerations, we need a minimum of input data for evaluation — what you already have on hand.
The pilot logs show where the model errs most often, and the reference answers show what quality level is considered acceptable. Describe the task in free form: that is enough to start.
- reference set: 200–500 real queries with answers and "correct / incorrect" labels — on it we calibrate automatic evaluation by a second model and measure fine-tuning accuracy;
- pilot logs: raw dialogues, prompt versions, and model answers as users saw them;
- acceptance criteria: which errors are critical, which are acceptable, and in what share of cases;
- domain and scenario description: whom the model answers, to which questions, and what is prohibited in answers;
- comparison baseline: metrics of the baseline model or the previous fine-tuning version;
- data and access constraints: where the data is stored, whether anonymization is needed, whether external expert labelers can be engaged;
- examples of disputed answers: 10–20 cases without consensus — they set the labeling boundary.
Then we send a work plan: the composition of stages from reference set assembly to acceptance, control points at each step, and team roles. If something from the list is not at hand — say so directly: we collect and label part of the input data ourselves, as a separate stage.
