Implementation of Customer Request Topic Classification
Topic classification of requests — automatic determination of the subject of incoming customer request. Incoming stream of emails, chats, calls (after transcription) is distributed across topic categories without operator involvement.
Design of Topic Taxonomy
Before training the model, it's necessary to create the correct hierarchy. Common mistakes: too few classes (everything falls into "Other"), too many (500+ classes train poorly), blurry boundaries between classes.
Two-level hierarchy works well:
Technical Issues
├── Connection Problem
├── Slow Speed
└── Personal Account Errors
Financial Questions
├── Payment and Tariffs
├── Refunds
└── Debt
Contract Questions
...
First level: 5–15 categories — high recall. Second level: detail for routing to the right specialist.
Training the Classifier
Stage 1 — data collection: 200–500 examples per class from request archive with historical operator labels. Important: operators often label inaccurately → quality control stage for labeling is needed.
Stage 2 — baseline: TF-IDF + Logistic Regression. Trains quickly, interpretable, good for starting. Accuracy on typical tasks: 82–88%.
Stage 3 — BERT fine-tuning: DeepPavlov/rubert-base-cased. Accuracy: 90–95% with 500+ examples per class.
Stage 4 — LLM zero/few-shot: GPT-4o-mini with class descriptions — for new categories without retraining.
Handling Multi-Topic Requests
"My connection doesn't work and I want to change my tariff plan" — two classes. Strategies:
- Multilabel classification (sigmoid + threshold 0.5)
- Split by sentences, classify each
- Select primary + secondary class
Monitoring and Drift
Topic distribution changes during sales, seasons, incidents. Monitoring: chi-square test for deviation from historical distribution. Alert on significant deviation → reassess classifier relevance.