AI-based translation quality estimation system: QE without references

Why Quality Estimation breaks down without reference translations

Reference-based translation quality metrics rest on one assumption: every segment has a reference. In practice, it exists only for the control set, while the main flow—interfaces, documentation, marketing texts—goes without it.

As soon as the reference disappears, translation quality assessment stops being a measurement and turns into an opinion.

Then manual review kicks in, and it quickly becomes a bottleneck. A linguist goes through segment after segment: the speed is predictable—minutes per fragment—but the volume of content grows faster.

The assessment arrives days after publication, when edits have already been made, and no longer affects quality.

The second problem is disagreement between reviewers. One considers a term choice an error, another sees it as a stylistic liberty, and the final score varies so much that the signal drowns in the noise of a particular expert.

Only corpus annotation using a single taxonomy reduces the spread: categories instead of free-form comments, fixed severity levels, and calibration on shared segments where reviewers compare their assessments.

Next, the model learns on this annotation and assigns scores without references—reproducibly and with a known error margin.

We will break down how this is addressed: from the annotation scheme and inter-reviewer agreement metrics to controlling score drift after each fine-tuning sample.

What automated quality control provides in the workflow

Automated quality control turns proofreading from continuous into sampled. The model evaluates each segment against the source text, and the linguist opens only what falls below the threshold.

On working language pairs, this removes 70–85% of the manual review volume: editors' time goes to disputed places and glossaries rather than re-reading stable fragments.

The threshold is calibrated, not guessed. We run a labeled sample, compare machine scores with editor verdicts, and set the boundary so that missing a real error remains an exception.

Each language pair has its own threshold: languages with rich morphology and short interface strings behave differently.

A single rating scale is provided by open metrics from the COMET family—a standard in Quality Estimation. They are trained on human ratings and are therefore comparable across languages, teams, and vendors.

Instead of five different “good/bad” labels, the report gets a single number and a formal acceptance boundary—disagreements can be about substance, not taste.

In the workflow, this shows up in two things. First, localization release speed: the release does not wait for a full proofreading cycle but ships together with the product, because the list of problematic segments is known in advance.

Second, manageability: the average score per release and the share of segments below the threshold are compared with the previous run, and a drop of several points is visible before the text reaches users.

Quality Estimation: comparing approaches without references

Reference-free assessment is based on scoring: the model returns a confidence score for each segment, and the document-level verdict is the result of aggregation.

Segment-level mode catches local distortions—a swapped term, a lost negation, a wrong number—but does not see that paragraphs have stopped being coherent.

Document-level mode sees coherence and repetitions but blurs individual errors in the average, so trigger thresholds are configured separately.

The modes below differ not in architecture but in the amount of labeled data that must be maintained and the acceptable cost of a missed defect.

It is correct to compare modes on your own sample: 300–500 segments with manual assessment on a multidimensional scale in the spirit of MQM show where the model errs systematically.

In the hybrid, selection is stratified—risky segments go entirely to an editor, the rest enter a random 5–10% sample, and the share of missed defects is calculated from it.

Implementation stages: from data audit to system handover

Implementing reference-free translation quality assessment is not installing a ready-made model but a chain of work where each step relies on the previous result.

An unaccounted domain or biased annotation at the input later produces an estimator that confidently errs on entire language pairs, so the order of stages is strict.

1. Audit of historical translations. We assemble a corpus from work already accepted by the client, annotate it on a direct-error scale—meaning, terminology, formatting—identify domains, language pairs, and segment types, and remove duplicates and truncated lines.
2. Design of the assessment scheme. We fix what we consider a critical error and what is acceptable variability, encode the rules into annotation, and split the corpus into training, control, and holdout parts.
3. Model tuning. We train the estimator on pairs of “translation—editor verdict,” handle rare domains separately so that bias toward frequent topics is not transferred to the rest.
4. Threshold tuning. We select boundaries at which a segment goes to manual review: we calculate recall and precision on the control part and balance editor workload—too narrow a threshold floods them with work, too broad a one lets meaning errors through.
5. Validation on a holdout set. We run the system on segments that did not participate in training, with stratification by domains and language pairs, compare with editor assessments, and review every disagreement in the upper part of the range.
6. System handover. We deliver configurations, the annotation policy, the retraining scheme, and the corpus update procedure, and train the client’s team to run the assessment and read reports independently.

After handover, drift control remains: once a quarter we compare scores with manual review, update annotation for new domains, and retrain the model. Describe the task and the volume of historical translations—we will come back with an estimate and a stage plan.

What we hand over: configurations and policies

We hand over not only a working system but also a set of artifacts by which it can be operated and developed without us.

The list below is typical for implementing translation quality assessment (Quality Estimation): it covers both day-to-day work and disputed situations when the automatic score diverges from the editor’s opinion.

• Assessment configurations — profiles for each language pair and content type: thresholds, feature weights, exception list. They are changed in a separate file, without editing code.
• Integration scheme — where the system connects to your translation pipeline: service call, batch file processing, export from translation memory. With exchange formats and behavior when the service is unavailable.
• Description of assessment rules — how the per-segment score is formed, which features influence it, what is disabled and why: the result stops being a “black box.”
• Disputed-case review policy — who reviews a disagreement between the automatic assessment and a human editor and within what timeframe, and how the decision returns to the settings.
• Reference segment set — a labeled sample on which the system is run after every threshold change: it shows whether it became more accurate or just different.
• Operating instructions — startup, metric monitoring, signs of degradation, and the procedure when quality drops.
• Decision and calibration log — recorded agreements on complex segments, the procedure for adding a new language pair, and the data on which the result is validated.

Case study: translation quality control on eight language pairs

Before: sampled control instead of a system

Eight language pairs and a constant stream of edits from editors and external translators—yet only a tenth of segments were checked.

For most pairs there was no reference, so quality was assessed manually: two editors independently assigned scores, and on the same segments the discrepancy reached two points on a five-point scale. Disputed cases went to a third round of agreement, and the release waited specifically for them.

What we set up

We built reference-free quality assessment (Quality Estimation) on COMET: the model predicts segment quality from the source—translation pair without requiring a pre-prepared reference.

The accumulated edits became the annotation—what an editor changed, the model learned to treat as a deviation rather than a variant of the norm.

For each language pair we selected its own thresholds: high-confidence segments are accepted automatically, borderline ones go to a human editor’s queue together with the score and a reason flag.

What happened to the metrics

The share of automatic decisions grew from a third to three quarters of segments, and the volume of manual proofreading decreased approximately threefold.

The spread of scores between language pairs narrowed: previously one pair consistently received inflated scores; now the scale is brought to a common form and is comparable across languages.

Release cycle time decreased from nine days to four—the bottleneck stopped being the agreement queue, and disputed segments became visible at the input rather than after proofreading.

What to do if you already have your own quality control deployed?

If quality control is already working, there is no need to start from scratch—the question is what exactly is missing: coverage of rare language pairs, signal freshness, or independence of assessment from your own team.

This determines the choice—extend the pipeline in-house or connect an external reference-free assessment.

The hybrid scheme turns out to be workable most often: your own rules and accumulated annotation remain the source of truth, while external assessment covers pairs and genres where you have few examples.

Both scales are reduced to one—otherwise the numbers cannot be compared and there is no trust in them.

Migration without stopping the process proceeds by running in parallel: the new pipeline assesses the same flow as the old one, results are compared on a shared sample, and disagreements are reviewed before switching. We go segment by segment—first low-risk language pairs, then the rest.

Extending the pipeline in-house is justified when the rules and annotation are part of your advantage and cannot be shared externally. Then the cost is the team’s time: the pipeline lives only as long as examples are fed into it.

Technical concerns: where automatic assessment errs

Reference-free assessment (Quality Estimation) predicts translation quality from the source—result pair without having a prepared reference.

On typical pairs and even annotation, the model maintains high agreement with an expert but drops where even a human assessor disagrees with themselves: rare language pairs, narrow terminology, long segments with reordered word order.

Errors cluster predictably: a fluent but meaning-distorted translation is overestimated by the model, while a literal but correct one is underestimated.

A separate category is proper names, numbers, and units of measurement: substituting them changes the meaning but hardly changes the wording, and assessment here is unreliable.

What to do with borderline segments?

Segments from the gray zone are not accepted automatically and not discarded but go to manual review. The share of such segments is a controllable parameter: a lower threshold means broader automation; a higher one means more accurate assessment.

Does the assessment hold when the domain or language changes?

When transferred to a new domain, agreement drops, and that is expected. Before running it in the flow, we recalibrate on a small labeled sample from the client, and we fix agreement with the expert as an acceptance criterion.

Let's discuss the task: how a quality assessment system is built

The first conversation is technical, without presentations: we need to understand where text appears in your flow and who currently decides on its quality manually. Nothing needs to be prepared in advance—it is enough to describe the process and show examples of translations.

Based on the conversation, we come back with an implementation plan and timeline estimate, not general promises.

To calculate timelines based on your data rather than market averages, we need a minimum of inputs:

• 2–5 thousand source—translation segments in a working format: xliff, tmx, or a simple tsv from your translation system.
• quality rules and error annotation, if it was maintained: guidelines, scale, examples of disputed cases.
• one or two assessors ready to annotate 300–500 segments: based on their verdicts we calibrate the model and calculate disagreements.
• description of the flow: at which step a person currently proofreads the text, where the decision is made, and what happens to it next.
• environment constraints: what data cannot leave your infrastructure and in what mode the assessment must operate.

In return, you receive not a report but a working plan: implementation stages with a point at which you decide whether to continue, and a pilot protocol on your data—the share of matches with the assessor’s verdict, a breakdown of disagreements by error type, and an assessment configuration for your language pairs and scale.

If some data is missing, it is not a blocker: we will assemble and annotate the corpus ourselves, but then a preparation stage will be added to the timeline. Describe the task in this form—we will come back with a work plan and estimate.