Request classification with AI: where routing breaks
Manual sorting of requests rests on one assumption: an operator has time to read the incoming flow and distribute it into categories before new ones arrive. While it is dozens of requests per day, the scheme works.
At a flow of several hundred, it falls apart — the first-line queue grows faster than it is cleared, and by evening some requests remain without a category and owner.
The first breakdown is request priority. In manual mode, it is set by a subjective criterion: who wrote louder, who called earlier, whom a manager asked to prioritize.
In service management practice, priority is calculated from impact and urgency, as described in ITIL — in manual triage, this logic relies on the operator's memory and falls apart under peak load.
The second breakdown is request routing. A manually selected category often goes to the wrong group, and the request gets bounced between lines, losing context: the client repeats the story, attachments are sent again, and time to first response doubles.
Major incidents drown in the general flow of similar questions — not because they were missed, but because the queue does not distinguish important from high-volume.
We break down how this is addressed: where exactly routing breaks, which request attributes need to be extracted, and how AI-powered request classification removes manual sorting from the first line.
What changes after implementation: response time and share of manual triage
The effect of AI request classification is visible in five first-line metrics: time to first response, share of requests routed to the right group on the first try, share of manual sorting, number of repeat escalations, and behavior at peak. We capture them before launch and after — from processing logs, not from employee perceptions.
First line: speed, accuracy, operator relief
The model labels the incoming flow by topic, urgency, and required expertise before a request reaches a human.
Response time to a request drops from hours to minutes: the route and priority are set at intake, the operator does not sift through the entire queue but immediately opens what is assigned to them.
For stable categories, classification accuracy stays around 90%; ambiguous cases go to retraining with labeling of edge examples.
The share of manual sorting is reduced severalfold: the dispatcher stops distributing the flow and handles only mixed topics, requests without clear text, and conflicting wording. The freed-up time goes to substantive responses, not routing.
Adjacent processes: load, escalations, analytics
Peak support load ceases to be a failure point. The classifier maintains constant throughput, so with a tenfold increase in flow, the queue does not grow proportionally — only the share of requests requiring a human response increases. Priority separates the queue: urgent ones move ahead, typical ones receive a ready answer.
A side effect is manageability. A breakdown by topics and causes appears, showing which failures generate requests and where repeat escalations accumulate. This effect extends to the product, not just the first line.
Classification options: rules, learning on labeled data, hybrid
The three approaches are not either-or alternatives but different points on a scale: rules and templates provide a predictable start, while learning on labeled data provides resilience to phrasings no one anticipated.
Two factors determine the choice: how many labeled requests have already been accumulated and how often the taxonomy of reasons changes.
| Option | When it fits | Limitations |
|---|---|---|
| Rules and templates | Few reasons, stable wording, need a predictable start without labeling | Breaks on paraphrases and typos; rules multiply and begin to conflict |
| Learning on labeled data | There is a history of requests and a clear class schema | Requires consistent labels; when the taxonomy changes, relabeling is needed |
| Hybrid classification scheme | The flow is heterogeneous: some cases can be formalized, some cannot | Both parts need versioning and regular quality control of labeling |
What each approach delivers in the short term
Rules provide a working route within days: engineers describe conditions, the classifier immediately routes requests, and errors are visible and fixed without retraining.
A hybrid behaves the same at the start, but the rules layer collects statistics from the very beginning — which requests the model labels with low confidence and what should be sent for labeling first.
In the long term
Learning on labeled data wins where the flow grows and wording changes: the share of correct routes is maintained through new examples, not new conditions. Here labeling quality is decisive — consistency of labels among annotators and regular review of ambiguous classes. Otherwise the model reinforces others' errors, and accuracy degrades imperceptibly.
Implementation stages: from flow analysis to handover to operations
We start not with the model but with the flow: how many requests arrive, through which channels, what share goes to the wrong place, and how much time operators spend on manual sorting.
These numbers show where classification delivers an effect and where a process regulation is enough.
- Request flow audit: export for a period, distribution by channels and topics, measurement of routing error rate and time to first response.
- Class and priority schema: list of topics and escalation rules. There should be few classes and no overlaps — otherwise the system gets confused at the boundaries.
- Labeling preparation: label historical requests according to the schema, set ambiguous cases aside. Here it also becomes visible which classes are missing in the data.
- Configuration: select the confidence threshold at which a request goes to an operator rather than to an auto-response. An error on an urgent request is more expensive than an extra manual check.
- Routing regulation: who and when reviews classes, how an operator disputes the system's decision, and where their correction goes.
Validation on historical data
We run the system on requests from a past period where the route is known and look at quality by each class, not overall accuracy.
We separately check urgent ones: the share of missed critical requests should be close to zero, even at the cost of manually sorting some ordinary ones.
Handover to operations
We hand over the configured system and documentation: class schema, thresholds, error review procedure. We train operators and support — how to read the system's confidence, what to do when the subject matter of requests changes. We remain available during the first period after launch.
What remains with the client's team: schemas, rules, labeling
It makes sense to accept the system not based on answers on a test set, but based on what remains within the client's environment and allows them to maintain it independently.
We hand over documents, labeling, and data handling rules — in their original form, with the right to edit them for your processes.
Documentation composition
- Class and priority schema: list of categories, boundaries between similar classes, and routing rules — which priority and recipient each category has.
- Labeled request set: a reference sample with consistent labeling, including ambiguous cases with documented rationale. It is used to check quality after any change.
- Rules for labeling new data: what to do with a request outside existing classes, when to introduce a new one, and who approves it.
- Maintenance regulation: adding classes, frequency of control runs, threshold for sending to retraining, responsible parties for each step.
- Description of input fields: what is fed as input, what is ignored, how attachments are processed.
- Decision log: where the classification history with priority and confidence is stored and how to use it to review erroneous triggers.
Data access rules
- Role separation: who sees source texts, who sees only anonymized exports, who changes labeling.
- Anonymization procedure: which fields are masked or deleted before entering labeling and logs.
- Storage and transfer: timelines, the person responsible for the environment, procedure for exporting the model and labeling when changing contractors.
Case study: how ticket routing became predictable after launch
The flow of requests came from email, chat, and telephony into one queue, and then first-line operators manually decided who the ticket was addressed to.
Priorities and service levels were described according to ITIL, but the labeling of the incoming flow relied on people's memory — and broke first as load grew.
What it was before
Tickets went into a common pool: the operator determined the topic, assignee group, and priority from the text of the email.
About 71% of requests were routed correctly on the first try; the rest were bounced between groups, and each transfer added a new waiting cycle to the response — the median time to first response stayed around four hours, and for urgent ones — up to two.
Manual triage took about 40% of first-line time: 3-5 minutes per email, and during peaks the queue simply grew.
Some failures were predictable: the topics "payment" and "access" were confused with each other, and high-priority incidents lined up in the common queue next to consultations.
What changed after
We labeled the request archive together with the client and trained the model on that labeling: it assigns a topic, group, and priority to a request, and passes ambiguous low-confidence cases to a human and adds them to retraining.
The decision is logged together with the text and features, so the cause of an error is visible without manual reproduction.
- correct routing — 94% vs. 71%
- median time to first response — 40 minutes, for urgent ones — 12
- share of requests under manual triage — 9% instead of 40%
- service-level breaches — 2% instead of 11%
How does rule-based request classification differ from a model?
Rules give a deterministic answer: if there is a contract number from the list or a word from the dictionary, the request goes to the right queue the same way on the first and the thousandth run.
The price of predictability is fragility: if a client writes "the money left but the goods are not there" instead of "payment did not arrive," the dictionary will miss.
Hundreds of expressions have to be maintained manually and run against the request archive with every edit.
The model solves the inverse problem: it generalizes wording, transfers experience from frequent examples to similar rare ones, and survives typos, transliteration, and channel changes.
The weak spot is the long tail: a class that appears only a few times in the labeling has nothing to learn from, and its quality degrades.
The model's answer is a probability, not an instruction, so the question "why is this spam" cannot be answered without feature attribution.
In practice, a two-tier scheme wins. Hard rules cover what cannot be left to statistics: legally significant triggers, blacklists, mandatory routing of large clients — they fire instantly and are explained in one line.
A model based on scikit-learn or a neural classifier takes the rest: topic, sentiment, signs of urgency and recurrence.
Explainability consists of two parts: the reason code from the rule and the model's confidence. Below the threshold, a request goes to an operator as doubtful rather than to an automatic queue — this is the protection against silent errors.
Operator corrections return to labeling and go to retraining once every few weeks, so the share of rare classes gradually grows while predictability remains on the side of rules.
Technical concerns: labeling, rare classes, quality control
Three questions come up in every assessment: how many examples to label, what to do with classes that occur rarely, and how to notice that classification quality has declined. We answer specifically — for these objections we have proven schemes tested on a live flow of requests.
How many requests need to be labeled at the start?
The volume depends on the number of classes and how semantically close they are.
We start with several thousand examples covering each class, then supplement with active selection: ambiguous requests where the model is uncertain go to labeling — the quality gain per unit of effort is higher than with random sampling.
What to do with rare request types?
A class that accounts for fractions of a percent of traffic cannot be improved by general training. A combination works: class weights, separate confidence thresholds for each type, merging similar types into a superclass with subsequent splitting. And the metric is recall for the rare class: in overall accuracy, its failure dissolves.
How to tell that classification quality has declined?
We monitor two signals: the divergence of the incoming text distribution from the reference sample and the share of the "other" class together with the average confidence level — they react to changes in wording earlier than operator complaints.
Every update goes through a regression set with reference answers, and we review ambiguous cases using an error matrix with annotator arbitration. If recall for a critical class drops, we roll the model back to the previous one and investigate the cause.
Let's discuss the task and come back with an estimate and work plan
To start the discussion, an export of requests for several weeks in the form in which they sit in the system is enough: text, channel, topic, final assignee, labels set manually by operators.
The less the export is cleaned up, the more accurately the real class distribution and the places where manual routing breaks are visible.
It is useful to add to it a reference of categories and priorities, a regulation of response times for each level, and a dozen ambiguous cases — those whose class operators still argue about.
- export of requests with text, channel, and processing history;
- current reference of categories, subcategories, and priorities;
- regulation of response and escalation times by level;
- examples of ambiguous and incorrectly routed requests;
- list of intake channels, including call transcripts;
- data restrictions: what cannot be moved outside the perimeter and which fields are personal;
- current routing scheme: who assigns the assignee and by what rule.
Based on this picture, we will analyze the flow, estimate the share of requests with an unambiguous class, and come back with an implementation plan: labeling preparation and model training on your examples, a pilot launch on one channel with parallel manual control, quality metrics and error review, then handover to operations with monitoring and a retraining regulation. Describe the task — we will come back with an estimate and work plan.
