Why manual review of operator chats does not scale
An operator closes dozens of dialogues per shift, while a supervisor manages to review a fraction of a percent of them. One review takes 8–15 minutes, that is, 15–20 conversations per day with a flow of thousands.
Conversations in Zendesk and similar systems are stored in full, but are checked selectively: automation measures first response time, while compliance with substantive criteria — greeting, identifying the need, correct transfer to the next department — remains with a human.
Selective listening to conversations turns into a lottery: a systemic error in the script or in transfer between lines lives for months until it lands in a random sample.
Added to this is the mismatch of review criteria — two supervisors give the same dialogue different scores, with the spread reaching tens of percent.
The operator receives contradictory signals, does not understand which standard to aim for, and the client sees the error earlier than the supervisor.
Feedback delay finishes off the effect: the review arrives several days later, when the context is forgotten and the behavior has already become established.
The supervisor's workload grows linearly with headcount — twice as many operators require twice as many reviewers, while coverage still remains a fraction of a percent of the total volume.
We break down how this is solved: how automatic evaluation of operator chats works according to unified criteria and what it changes in shift operations.
The effect of continuous review: speed, completeness, unified criteria
Continuous review of dialogues moves quality control from sampling mode into streaming mode: every conversation is evaluated, not random 5–10% of it. Markup follows a formalized rubric, so the result does not depend on which senior operator opened a particular chat.
The evaluation is performed by a language model — in such projects we rely on OpenAI and solutions of comparable quality, not on a single hard-coded engine.
Full sampling instead of a random ten percent
- Coverage — 100% of dialogues versus 5–10% with manual review; a conflict chat no longer falls out of control because it did not make it into the reporting week.
- Early problem detection: the share of incidents that surface through a customer complaint rather than through conversation review falls from 40–50% to single-digit percentages.
- Statistics accumulate quickly: a bottleneck in the script is visible on 20–30 dialogues, not after reviewing hundreds of similar requests.
Review speed and a unified criteria scale
- Conversation review speed — from 10–15 minutes of manual reading to 1–2 minutes to check the draft evaluation and confirm it.
- Consistency: divergence between reviewers on disputed dialogues is reduced from 25–30% to 5–7%.
- Disputed cases go into a separate queue with the reason for doubt indicated, rather than being rewritten from scratch “by eye.”
Overall, the class of errors changes: the problem is recorded within the dialogue, not a month later in a churn summary. That is why next we break down how the rubric works and how it is updated without retraining the entire system.
Architecture options: rules, ready-made model, fine-tuning
The choice is between four classes of solutions, and it determines the volume of markup, the frequency of retraining, and how the system behaves when operators start writing differently.
| Option | When it fits | Limitations |
|---|---|---|
| Rules and dictionaries | There are formal requirements: greeting, mandatory phrases, prohibited language | Breaks on paraphrasing; policy edits are made manually |
| Ready-made model without fine-tuning | A quick start and general assessment of tone and politeness are needed | Does not know your criteria, makes mistakes on domain terms |
| Fine-tuning on your markup | There are several thousand labeled dialogues and stable criteria | Requires regular retraining when scripts and products change |
| Hybrid scheme | Most real implementations | More expensive to maintain: both parts and a reconciliation layer are needed |
Rules and dictionaries: predictable, but fragile
Rules are fully explainable: for each score, it is clear which condition triggered, so they are easy to defend before the operator and quality assurance.
But language is not limited to templates — change the wording of a script, and the dictionary begins to produce false positives, while edits are made manually and transfer poorly to adjacent topics.
Fine-tuning on your markup and limits of applicability
Fine-tuning transfers your criteria to the model: it learns from expert assessments, not on abstract politeness, and confidently handles wording that is not in dictionaries.
The limits are visible immediately — the model inherits markup biases, so high scores for “own” operators will enter the weights. A reference sample and validation on new data are needed, otherwise quality quietly declines as products and scripts change.
The hybrid scheme wins because it separates tasks: rules cover formal checks, the model covers meaning and context, and the reconciliation layer resolves discrepancies and sends disputed dialogues for manual review. Such a loop provides both explainability of the score and resistance to paraphrasing.
Implementation stages: conversation audit, scheme, configuration, acceptance
An audit of dialogue history provides the factual basis: volume of flow, share of escalations, repeat contacts, typical topics. On this data we fix the criteria scale and only then move to configuration.
- Audit of dialogue history: we export conversations, label topics, message length, escalations, and repeat contacts.
- Approval of the criteria scale: for each violation — wording, weight, and “yes/no” examples, so disputed cases are resolved consistently.
- Markup scheme: what we consider the unit of evaluation — a dialogue or an individual message, who labels, and how markup quality is checked.
- Configuration of the evaluation model: training on the labeled corpus and calibration of trigger thresholds on borderline dialogues.
- Acceptance on a reference sample: a run on dialogues the system has not seen, and comparison with mentor assessments.
- Handover and team training: review of reports with supervisors, procedure for disputing a score, and fine-tuning on new examples.
How the reference sample is formed
We assemble the sample stratified: for each topic we take dialogues in proportion to their share in the flow and separately add rare but critical cases — conflicts, complaints, policy violations.
Each dialogue is independently evaluated by two mentors, and discrepancies are reviewed at a reconciliation session. This prevents fitting the model to convenient examples.
What counts as successful acceptance
Acceptance passes if the system reproduces mentor assessments with consistency no lower than between the mentors themselves, and does not miss critical violations.
We separately monitor the share of false positives: a supervisor should review the flags in a reasonable time, not drown in them. We record the outcome in a protocol — what works, what we are still tuning, and who is responsible for revision.
Handover package: markup, rules, policies, reporting dashboard
After launch, evaluation of operator chats no longer depends on who exactly is reviewing dialogues today.
We hand over the documentation package to your repository: rules, markup, and settings sit next to the system and are changed by your team, without contacting us for every clarification.
- Markup rules: a list of criteria with “normal / violation” wording, the scale, disputed cases with an explanation of how to interpret them.
- Labeled dialogue corpus: reference conversations with assigned scores — the system is configured on them, and new rules are also checked against them.
- Evaluation configuration: criteria weights, automatic review threshold, list of stop phrases and exceptions — in files, without editing code.
- Reporting dashboard: cuts by operators, shifts, and request topics, score dynamics by week, export of a sample for manual review.
- Access matrix: roles, view and edit permissions, procedure for granting and revoking.
- Policies for reviewing disputed dialogues and revising criteria with owners and deadlines.
Policy for reviewing disputed dialogues
A score the operator or manager disagrees with is not deleted or edited manually — the dialogue goes into the second-line queue. A senior labeler reviews it according to the same rules, the decision is recorded with justification, and it replenishes the corpus as a new reference.
Recurring disputed cases become a separate rule item — the share of disputed scores decreases from iteration to iteration, and with it the time for manual reviews.
Access matrix and role separation
An operator sees their dialogues and final score, a shift manager sees their department and the reasons for a score reduction, an analyst sees an anonymized sample without personal data. The administrator manages rules and settings but does not evaluate dialogues manually.
Rights are granted by roles from your user directory, access changes are logged, and export of scores outside the perimeter is closed.
Case study: reduced review time and eliminated criteria discrepancies
Before implementation, quality control relied on manual sampling: a supervisor reviewed 5–7% of conversations per shift, and one conversation took 12–15 minutes.
The criteria lived in the policy and in the reviewer's head, so two scores of the same dialogue diverged in about a third of cases. Feedback reached the operator after two or three days, when the conversation was already closed and it was too late to change behavior.
What the first weeks after launch showed
For the first week, the system ran in shadow mode: it evaluated the entire flow, but scores did not go to operators.
The markup was compared with the conclusions of three supervisors and connected to the existing request system (Zendesk) using standard tools, without touching workstations.
Some of the discrepancies turned out to be not in the model but in the policy itself — the criteria wording was refined and reinforced with dialogue examples.
| Metric | Before | After |
|---|---|---|
| Dialogue coverage | sample 5–7% | entire flow |
| Review of one conversation | 12–15 minutes | only disputed cases |
| Score discrepancies | a third of cases | single-digit percentages |
| Feedback to operator | 2–3 days | within an hour |
Disputed scores are no longer lost: the operator sees the score next to a quote from the dialogue and a link to the criterion and can dispute it; then the supervisor decides.
Only appeals are reviewed manually — a fraction of a percent of dialogues instead of continuous sampling, and the freed-up time goes to training those whose scores dropped the most.
What to do if the quality evaluation policy is already approved?
The existing policy remains the source of truth, not a draft that must be rewritten for the system.
Criteria, scales, and wording are transferred as is: the criteria tree, their weight, and branching rules repeat the approved document, including disputed situations and exceptions — they become separate rules rather than getting lost during automation. Internal instructions for reviewers remain in force; only the tool they use changes.
The transition from manual review goes through a parallel run. The system evaluates the same dialogues as the reviewer, but the first iterations do not affect anything: we review discrepancies manually, and they show which criteria wording is read ambiguously.
Usually after two or three such cycles, consistency of scores with experienced reviewers becomes sufficient for automatic evaluation to become primary, while a human moves to selective control.
The system integrates into current processes without changing the operator's entry point: dialogues are pulled from where they currently reside, and the evaluation result is returned to the same place — the dialogue card, report, or internal portal.
Low-confidence scores go into a manual review queue, so the reviewer deals with exceptions, not the flow.
The policy lives and changes, so criteria are not fixed forever. We make an edit in the criteria set, run calibration on the labeled sample, and if necessary fine-tune the model.
Policy versions and criteria set versions are mapped to each other, so old scores remain reproducible and comparable across periods.
Technical concerns: integrations, access, markup, resilience
Before launch, the engineering side checks four things: integration with working communication channels, privacy and access, resilience to live language, and behavior when request topics change. We answer briefly, without generalities.
Evaluation of operator chats reads conversations in read-only mode and stores results in a separate storage — the operator's workflow does not change. Rights are inherited from the working system's roles, and every access to dialogues is recorded in the log.
Do we need to modify the working conversation system?
As a rule, no: we connect through standard exports or an event stream. If there are no exports, we install an intermediary module that retrieves dialogues and returns scores — the operator interface remains the same.
What about privacy and access rights?
Personal data is anonymized before entering the model: names, phone numbers, and order numbers are replaced with labels. Access to source texts is granted by roles, with an action log.
How does the system handle slang, typos, and mixed speech?
Evaluation is based on meaning, not word matching, so abbreviations, typos, and Latin characters inside a phrase do not break the review. Disputed dialogues go to manual review and fine-tune the model.
What happens when request topics change and volume grows?
New products and request types require updating criteria and several examples in markup — there is no need to rebuild the entire solution. Volume grows without loss of speed: dialogues are processed in batches and archived according to a specified retention period.
Let's discuss the task: what to bring to the first review
To turn the conversation into an estimate of scope rather than an exchange of expectations, it helps to bring a specific set of materials to the first meeting. There is no need to prepare anything in advance — what already exists in your working systems is enough.
- Historical dialogues for 2–4 weeks broken down by channel: website chat, messengers, email. This is the basis for a trial run on your data, not on a demo sample.
- List of criteria currently used to evaluate operators manually: script compliance, tone, completeness of answer, first-contact resolution. The markup scheme is built from them.
- Examples of supervisor scores — at least a hundred dialogues with scores and comments, to compare the model's logic with your quality control team's opinion.
- Evaluation policy and rules for escalating disputed cases: discrepancies on conflict dialogues matter more than formal criteria.
- Request routing scheme: where dialogues come from, where they are stored, whether export and anonymization are possible.
- Personal data restrictions: what cannot be taken outside the perimeter, whether masking of numbers, addresses, and payment data is required.
- Roles on your side: who reviews disputed scores and who accepts the result.
Next we look at what share of dialogues is labeled unambiguously, where criteria conflict with each other, and return with a plan and timeline for the pilot: sample size, share of matches with supervisor scores, points of mandatory manual review.
Describe the task in a convenient form — we will discuss it substantively and return with an estimate of scope.
