Why an AI HR chatbot stalls at pilot launch
A pilot launch of an assistant usually looks convincing: on prepared questions, an AI HR chatbot answers smoothly and quickly.
Problems start in the second or third week, when employees come with real-world wording — "how many vacation days do I have left," "what about sick leave," "can I transfer to another department." That’s when both an incorrect answer and a silent handoff of the dialog to an HR employee surface.
The first typical cause is that the assistant answers based on the wrong data — not the data the company actually runs on. Regulations are spread across different files, orders have been updated several times, and some HR information is available only in the record system.
If the knowledge base is assembled manually once, the model confidently quotes a revoked version — and that is worse than no answer at all.
The second is that there are no rules for when the assistant must stop. Without explicit routing by question type, it either tries to answer everything, including personal cases, or hands almost every dialog to a human.
In the second scenario, automating HR requests does not reduce the workload on the HR service: the same questions, only now marked "came from chat."
The third is that no one measures quality. Feedback like "answer useful / not useful" does not show what share of requests the assistant got wrong, and where it simply stayed silent.
Without a set of benchmark questions and analysis of real dialogs, the pilot cannot be fixed and cannot be defended before management.
We break down how this is closed: from preparing sources and escalation rules to measuring accuracy on a sample of real requests.
Measurable effect: response time, workload, data quality
The first effect is visible in response time for the employee: a typical question about vacation, a certificate, or a schedule is resolved in a dialog in seconds, not after hours of waiting in the HR service inbox.
We separately calculate the share of automatic answers — on a mature corpus of regulations, this is 60–75% of requests, while the rest reach a specialist with context already assembled: department, request history, and a link to the relevant clause of the document.
The second metric is first-line relief. The load on operators drops by a multiple and its nature changes: instead of uniform answers, they handle non-typical cases where judgment is needed, not a retelling of the regulation.
The assistant does not "hide" itself: the employee always sees when the system is answering and when a human joins in — and can pass the dialog onward with one button.
We do not tie the solution to a single model provider and work in the OpenAI class, so replacing a model or adding an in-house one does not break the logic of scenarios and metric tracking.
The third metric is the quality of data in requests. Each dialog is automatically labeled by topic, department, and reason for the question, so the department head sees which topics are growing and which internal documents generate the most clarifications.
This feedback goes to the regulation owners, and the next revision of wording removes some questions before they are even submitted — while the share of automatic answers grows without modifying the assistant itself.
We capture all three metrics from the first day of operation and compare them monthly against the baseline measurement before launch: it is clear what exactly produced the shift — expanding the knowledge base, rewritten scenarios, or new entry points.
Assistant operating modes: knowledge base, generation, hybrid
The three operating modes of the assistant differ not cosmetically: each defines where the fact in the answer comes from. From the company’s indexed documents, from the language model, or from a combination.
This determines whether we show the employee a link to the regulation clause and what happens when a question is not in the knowledge base.
The mode is chosen based on the cost of an error in the scenario, not on taste. For orders and HR procedures, traceability to a specific document is required; for draft announcements and letters, freedom of wording is required. That is why the modes coexist in one bot, and routing is by question type.
| Option | When it fits | Limitations |
|---|---|---|
| Knowledge base search | Reference questions: regulations, HR procedures, conditions | Answers only with what is in the index; for a question with no matches, returns "not found" |
| Answer generation | Drafts of announcements and letters, rephrasings | No grounding in documents: a plausible but incorrect norm; source is not shown |
| Hybrid answer mode | Main HR scenario: search first, then wording with a link to the fragment | More complex setup: relevance threshold, refusal rules, regular output review |
Switching between modes is controlled, not random: if the retrieved fragments match the question poorly, the assistant does not make up the rest of the answer, but asks to clarify the wording or passes the dialog to an HR specialist. This keeps the hybrid mode predictable even for questions that are not yet in the knowledge base.
Implementation stages: from process audit to handover to support
Implementing the assistant is a sequence of verifiable steps, not connecting a model to chat. Each stage ends with an artifact: a labeled corpus of requests, an integration diagram, a test run report. Below are the assistant implementation stages from audit to handover to support.
-
Request audit. We export history from the ticketing system, email, and HR records, and group it by topic: vacations, certificates, onboarding paperwork, calculations. The sample shows what share of requests is typical and where access to personal data is needed.
-
Integration diagram. We determine which systems the assistant accesses for facts and differentiate access by role — an employee receives only their own data. We separately describe the points where a dialog is transferred to a live specialist.
-
Knowledge base setup. We split regulations into fragments and link them to topics, and set a rule: answer only from confirmed sources; when confidence is low, do not guess.
-
Run on real requests. We take a held-out sample of requests and compare the answers against the HR service’s benchmark. We separately count the share of cases where the assistant correctly declined to answer.
-
Pilot group of employees. 20–30 people from different departments use the assistant in normal mode. The dialog log shows which wordings are not recognized and where the load on people grows.
-
Refinement and handover to support. We correct wording, add scenarios, and refine thresholds for transfer to a human. The HR team receives a panel with dialog history and a regulation for updating the knowledge base — ongoing support then follows it.
What we hand over: configurations, regulations, documentation
An HR assistant is not only a working service, but also a set of artifacts by which your team can support it, further train it, and hand it over to another contractor without code archaeology.
Everything listed is documented in your repository and knowledge base, not left in the implementers’ chat history.
- Assistant configuration: system instructions, scenario tree, confidence thresholds below which an answer is not returned to the user — with change history and rollback to the previous version.
- Integration diagram: which systems are connected (HR records, calendar, knowledge base), in which direction data flows, where failure points are, and how unavailability of an external service is handled.
- Access rights matrix: roles for employee, manager, HR specialist, and administrator; which data and actions each is allowed; which fields are masked in the answer.
- Escalation regulation: conditions for transferring a dialog to a human, the queue and response times, who reviews unsuccessful answers, and how they enter refinement.
- Scenario descriptions: the list of topics the assistant handles on its own, the boundaries of its competence, and prepared answers for borderline questions.
- Support instructions: how to find dialog history, reproduce a failed answer, disable a specific scenario, and restore the previous configuration.
- Data handling policy: what is stored, for how long, how it is deleted upon employee request — this is sufficient for internal audit and inspections.
We conduct the handover as a separate stage: we review the artifacts with your engineers and support team and answer each item. After that, ongoing support can be handled in-house.
HR service case study: metrics before and after implementation
The client is a manufacturing company with sites in several regions: about 4,000 employees and an HR service of nine people.
The request flow consisted of typical questions about vacations, schedules, certificates, and document processing; they came to a shared inbox and a work chat, a response took on average a working day, and more than half of the requests were repeated verbatim.
Metrics before and after
The Кадровый помощник assistant was connected to the internal regulation base and document templates; disputed topics were left to a live specialist. Measurements were taken on the same request flow a quarter after launch.
| Indicator | Before | After |
|---|---|---|
| Average response time for a typical question | 6–8 working hours | 2–3 minutes |
| Requests closed without HR staff involvement | 12% | 68% |
| Share of recurring requests in manual work | about half | about one fifth |
| Preparing a certificate from a template | 25–40 minutes | 3–5 minutes |
I will separately note the class of failures that disappeared: the employee no longer waits until the end of the day for an answer, and a complex case reaches the HR specialist with context already assembled.
What remained outside the project scope
- The assistant prepares draft documents but does not sign or process orders.
- Questions about a specific employee’s pay, disciplinary actions, and dismissals go directly to a specialist.
- Rare topics are only partially covered: an answer without grounding in a regulation is marked as unverified.
- Exchange with an external HR portal remained the client’s responsibility.
What to do if you already have your own chatbot
An existing assistant almost never has to be thrown away entirely. It usually has three layers: channels and connections to record systems, scenarios with branching logic, and the knowledge base.
The first layer is migrated almost unchanged; the second and third are rewritten for a model that answers with text rather than through a button tree.
Integrations remain as they are: access to the HR system, calendar, ticketing system, corporate messenger such as Microsoft Teams, and employee rights and roles.
Webhooks, service accounts, and access differentiation are preserved — only the entity that generates the answer changes.
The logic is described anew. "If — then" branches turn into instructions for the model and knowledge base search; to keep answers consistent with company policies, documents are split into fragments with metadata — department, scope, access level, effective date.
We do not lose accumulated scenarios: dialog logs are analyzed by frequency and wording, and the top real questions become a regression set.
The new assistant runs in shadow mode for some time — answering in parallel — and discrepancies with the old one are reviewed before the switch, so there are no breaks in familiar scenarios.
Describe which bot is running now, which channels it covers, and what has stopped working for you — we will come back with a migration plan and estimate.
Technical concerns: access, privacy, answer quality
The assistant does not get access to the entire HR system.
We create a separate service account with read-only rights to the necessary fields and restrict it to data from one department: a request about the vacation schedule will not pull information about compensation from adjacent teams. Every access to HR data is logged — it is clear who asked, what they received, and on what basis.
Sensitive data is not sent for model fine-tuning. Personal values are masked before being written to dialog history, the retention period is set by company policy, and export rights are granted by role.
Will the assistant get access to the entire HR database?
No. Access is granted on a minimum-necessary basis for answering: the service account sees only specific fields and only within its department. Expanding rights goes through approval and is logged.
How do you control answer quality?
We run a set of benchmark questions on HR topics before every knowledge base update. Answers below the accuracy threshold are reviewed, and disputed topics are passed to a live specialist.
What happens if the assistant makes a mistake?
Outside its domain, it does not improvise: it says it does not know and transfers the dialog to a support employee. There is also a kill switch — if the error rate grows, the function is disabled without affecting other services.
Let us discuss the task and come back with an estimate and work plan
To make the estimate for an AI HR chatbot substantive rather than a rough guess, the task review follows a single route: we look at request scenarios, data sources, and access restrictions, and then come back with a written document that records the estimate, pilot composition, and acceptance criteria.
Below is what is useful to send at the start; if something is missing, we clarify the gaps on a call with the relevant engineers.
- A list of real employee questions to HR — from frequent ones ("how many vacation days do I have left") to rare and disputed ones.
- Regulations, policies, and instructions the assistant should answer from: the more complete the corpus, the lower the share of "I don't know" answers.
- A list of source systems — HR system, HR portal, file storage — and who is responsible for access to each.
- Requirements for access differentiation: which topics an ordinary employee sees, and which only HR and the department manager see.
- Restrictions on personal data and internal documents: what the assistant must not show and must not write to logs.
- Acceptance metrics: for example, the share of answers without transfer to a live specialist and accuracy on a benchmark question set.
- A contact person on the client side who confirms scenarios and accepts the pilot.
As a result, you receive a work plan with stages and timelines, the pilot composition (which channels we connect — chat in the corporate messenger, a widget on the portal), an access differentiation scheme, and a set of checks that will make acceptance objective. Describe the task and send what you have — we will come back with an estimate and plan.
