Why classification of customer requests breaks at scale
At a flow of hundreds of requests per day, classification relies on people: a frontline operator reads the request, manually determines the topic, and moves it to the right queue.
While there are few inquiries, the scheme works — there are a dozen topics, and a person keeps them all in mind, without a reference guide.
The bottleneck in request processing appears when the flow grows and topics begin to overlap.
The same request contains a payment question, a delivery complaint, and a request to close access — manual sorting of requests requires choosing one category, and the choice is made anew each time.
Decisions are made sequentially, one request at a time, so frontline throughput does not grow with the load.
Then accumulation kicks in: the frontline queue grows faster than the incoming flow, and some requests wait for triage longer than the response itself takes.
At peaks, misrouting sends a request to the wrong department — the customer receives an answer to someone else's question, writes again, and the original request settles in the correspondence history without an owner.
The first response time metric still holds, while full resolution time diverges from it by multiples.
We break down how this is closed: from corpus labeling and rules for splitting overlapping topics to automatic classification, which assigns topic and priority at the moment of arrival and leaves only disputed cases to a human.
What automatic request routing provides
The classifier parses a request at the input: determines the topic, language, urgency signals, and sends it to the right queue along with a draft response.
The share of automatic recognition on a closed set of topics stays in the range of 0.75–0.9 — the remainder is not "thrown in" at random but goes to the general queue with a note for the operator.
Confidence threshold is the main configuration parameter.
Below it, the system honestly says "I don't know" and passes the request to a human: false routing is more expensive than manual labeling because the customer receives an answer from someone who does not resolve the issue.
We select the threshold on a holdout sample of real requests, not on showcase examples, and revisit it as new topics appear.
First response time is reduced through automatic confirmation: the customer immediately sees the request number, the determined topic, and the expected response time.
Operators stop spending the frontline on sorting — the load is leveled between groups, and peak hours no longer hit a single duty operator.
A separate effect is that the class of lost requests goes away: those without an owner, accepted outside working hours, or resent after an intake failure. Every request gets a route and a deadline, and deviations from it are visible in the report.
We integrate this with the tracking system you already use — from Zendesk to a custom one over email and messengers.
Approaches to labeling: rules, dictionaries, learning from history
Labeling a request flow is not a choice between "rules or learning from data" but a decision about where the responsibility of dictionaries ends and statistics begins.
Below are three schemes that are actually chosen between at the start, and the limitations of each: they show up not in a demo but in the second or third month of operation, when the flow changes.
| Option | When it fits | Limitations |
|---|---|---|
| Rules and dictionaries | 5–15 stable topics, there is characteristic vocabulary: SKUs, service names, typical wording | Breaks on paraphrases and typos; the dictionary grows faster than the flow; rule conflicts are resolved manually with priorities |
| Learning from historical data | A history of requests with assigned topics has been accumulated, and customer wording is diverse | Requires verified sample labeling; when a new topic appears, retraining is needed; quality drops on rare classes |
| Hybrid labeling scheme | Rules cover explicit cases, learning covers the rest, disputed ones go to manual triage | Requires agreed confidence thresholds and feedback: error analysis returns to sample labeling |
The hybrid scheme gives the best balance: dictionaries remove the predictable part of the flow, learning from history handles paraphrased requests, and the confidence threshold decides what goes to an operator.
The loop is explainable at every step: you can see which rule or which example from history produced the topic, and this also simplifies expanding the set of classes without a full rebuild.
How implementation proceeds: stages from audit to handover
Request classification does not come down to installing a ready-made model: the set of topics, confidence thresholds, and routing rules depend on your flow. That is why we proceed in stages, and each is closed with a result that can be checked on your own data.
- Request flow audit — we export history from all channels, count volume, share of duplicates, short and mixed requests.
- Topic list — a working taxonomy of 15–40 first- and second-level topics, an "other" class, and a rule for requests that fall into two topics at once.
- Instruction and labeling — we fix labeling rules and handling of disputed cases so that labelers do not diverge from one another.
- Setup and training — baseline model variant, confidence thresholds per topic, separate handling of short requests and attachments.
- Pilot on a data slice — a run on a held-out slice that the model has not seen, and comparison with manual labeling for each topic.
- Acceptance by metrics — precision and recall by topic, share of "other", share of confidently assigned; we agree in advance on the threshold below which a decision does not go into production.
- Handover to support — taxonomy documentation, retraining and relabeling procedure, flow monitoring, and the procedure for actions when metrics drop.
What happens after acceptance
The taxonomy continues to live: new products appear, seasonality changes, and the request distribution drifts away from the training sample.
Once a quarter we revisit thresholds and retrain the model on newly labeled requests — the share of automatic routing holds steady rather than slipping.
Project artifacts: rules, schemes, revision procedures
After project delivery, the client's team is not left with a "black box": everything that determines the classifier's behavior lies in its repository and is versioned together with the code.
Without these project artifacts, the very first new request topic will require us — and support will pause.
- Integration scheme: where requests enter the classifier (CRM, support channel, queue), where the failure points are, and how retries are configured on failure.
- Access matrix: who sees labeling, who edits topics, who changes the confidence threshold — rights are separated by role, not "the admin has everything".
- Topic revision procedure: frequency (on schedule or by share of unclassified), procedure for adding a topic, and checking for overlap with existing ones.
- Operator instruction: what to do with an uncertain request, how to mark it, where it goes, and how quickly labeling enters retraining.
- Classifier configuration: confidence threshold, exclusion dictionaries, rules for routing disputed requests to manual review.
- Topic catalog with a synonym dictionary: what topics are called, what they include, and what wording customers use to denote them.
- Control sample of requests with expected topics — run after every change to see whether something old was broken.
- Changelog: what, when, and at whose request changed in the rules — without it, a disputed decision cannot be rolled back.
The documents live next to the model and are updated in the same order as code: edit — check on the control sample — entry in the log. The system can be developed by your own team, and only disputed areas can be handed to us.
Case: automatic routing share grew to 92%
Before implementation, routing relied on keyword rules and manual triage. About 40% of requests were routed automatically; the rest were distributed by frontline operators manually and could not keep up at peaks: the night queue lagged by 6–8 hours.
Every fifth transfer turned out to be erroneous — the request migrated between groups while the customer waited for a response.
From labeling to routing
First, we labeled the historical array of requests according to a unified taxonomy. Disputed cases were closed not by labeler voting but according to the Request Labeling Regulation: the rule was fixed once and then applied to the entire flow.
This raised label agreement from 78% to 96% — the model stopped learning from contradictory examples where the same wording was labeled with different topics.
What happened to the metrics
The scheme went into production with a confidence threshold: above the threshold, a request goes to the group automatically; below it, to the manual triage queue with a hint on probable topics.
The share of automatic routing grew to 92%, erroneous transfers fell from 17% to 4%, and the night queue lag decreased from 6–8 hours to less than an hour.
Resilience to peaks was tested separately: when the flow grew 2.5 times, the share of automation held at 89–91%, because the load falls on the short triage queue rather than on people.
Every update passes a regression check on a held-out set of historical requests — if precision on a rare topic drops, the version is not rolled out.
How does learning from history differ from rules and dictionaries?
Rules win where the wording is stable and the cost of error is high.
Contract number, account details, statement request, account closure request — this is a lexicon with a fixed dictionary, and a rule for it gives precision that the model does not reproduce: it either fired or did not, and the answer repeats from run to run.
While there are a dozen topics and the flow is homogeneous, dictionaries are enough — they are easier to explain to support and to use for analyzing a disputed case.
The model outperforms rules where the dictionary ends.
"Stopped arriving", "charged twice", "I want it like last time" — you cannot prepare enough rules for such wording, while history-trained triage follows them by meaning and is resistant to typos, transliteration, and two topics in one request.
The price for this is a probabilistic answer: the output is a distribution over topics, and the decision is made by the confidence threshold.
The hybrid approach is built on a separation of roles. Rules take short unambiguous routes and protected categories where an error is more expensive than abandoning automation, while the model takes everything left after filtering.
Requests below the threshold go to the operator queue, and it is precisely this queue that becomes the source of labeling for the next retraining.
Dictionaries are not eliminated but narrowed to the role of a reference and a safeguard.
Once a quarter we analyze what fell into "other" and where a rule fired falsely, and add wording — but as soon as one phrase fits two rules, it is more profitable to move the topic to the model. Dictionary revision becomes a regular procedure rather than repair of accumulated conflicts.
Labeling, quality, and revision: questions before the start
Before the start, the client usually checks four things: whether there is enough data and how to ensure labeling quality, what to do with overlapping and rare topics, how to connect new request channels, and whether the model can be trained on real dialogues. Below are direct answers, without generalities.
How many requests are needed so that labeling quality does not fall apart?
It is not volume that decides but labeler agreement: on disputed topics, two specialists diverge in 15–20% of cases, and this delta must be removed before training.
We include a labeling instruction with edge cases, a control sample, and an agreement measurement — otherwise the model will learn noise instead of the topic.
Topics overlap and are constantly revised — what then?
Overlap is normal: delivery and return live in one request.
We define priority rules and allow a request to receive a primary topic plus adjacent ones, and we conduct topic revision using an error log: we look at which pairs of classes are confused most often and refine the boundaries of wording.
Do rare topics sink in the general flow?
Quality drops first on rare classes, so we measure quality separately for each topic, not with a single average figure. We balance the imbalance with weights and additional examples, and mark rare topics for manual review — that way they do not dissolve into the mass ones.
How are new channels connected and what about personal data?
A new channel enters as a separate source with its own format, and we compare its labeling with the already accepted one — otherwise classification of customer requests will drift on unfamiliar wording.
Personal data is anonymized before training: phone numbers, addresses, and order numbers are replaced with labels, and access to source data is restricted.
Let's discuss the task and come back with an estimate and plan
The project start depends not on the model but on data and decisions that only your side can make. We do not need a ready-made clean export — bringing requests to a unified form and initial processing are on our side. What we really need from your side so as not to lose time in the first weeks:
- history of requests over several months in its original form: emails, tickets, chat dialogues, call transcripts — without manual cleaning and filtering;
- existing categorization, if any: list of topics, operator instructions, examples of disputed cases;
- a person responsible for the topic on your side — someone who confirms category boundaries and resolves discrepancies, otherwise the classification will diverge from your internal logic;
- access to the real contour of sources (email, request tracking system, help desk, messengers) with the same rights as operators;
- a list of topics where operators label confidently and those where they argue — this boundary determines where a human is needed in the output;
- restrictions on personal data and export outside the perimeter — the processing scheme depends on them;
- acceptance criterion: what share of correct hits is considered workable and on which sample we measure it.
Describe the task — on these points we will come back with a scope estimate and work plan, including the topic scheme, labeling procedure, and quality control method on your data.
