Why an AI agent on Stack AI doesn't get out of pilot
An AI agent pilot starts the same way: a dozen preselected questions, a carefully loaded knowledge base, Stack AI answers confidently — and the solution looks ready.
In a real process, the request comes from another system, in a different form and from a different user, and the agent already answers differently.
The first context gap appears where the demo got by with a flat list of documents. In a working scenario, the agent needs customer data, request history, and links between records from different systems.
Without them, it substitutes a fact that is similar in wording but incorrect, and the error surfaces on the first serious request.
The second reason is access rights. While the agent is alone in the pilot and works on test data, the question of roles does not arise.
When connected to internal databases, it either sees too much or cannot find what it needs — for the customer, this is no longer cosmetic but a risk of leakage.
The third is that there is no owner of changes. Regulations and the knowledge base live their own lives, wording becomes outdated, and no one is responsible for answer quality. The agent degrades unnoticed: first rare inaccuracies, then employees simply stop using it.
The bottleneck is almost always not in the model, but in the process around it. Therefore, AI agent implementation hinges on what data and access it receives at each step and who is responsible for the result. We break down how this is addressed at the architecture level.
What changes after managed agent implementation
Managed implementation starts not with a showpiece dialogue, but with fixing the metrics by which the agent will be accepted.
First of all, request processing speed changes: the agent itself requests the ticket status, history, and balances in accounting systems, assembles an answer, and gives the operator a ready draft. The cycle shrinks from dozens of minutes to one or two, and the person handles only disputed cases.
The share of auto-replies grows not by “allow everything,” but by an explicit confidence threshold and escalation rules.
We describe which types of requests the agent closes itself, which it hands over marked “check,” and which it immediately routes to an operator — with the full dialogue context so that the operator does not ask again.
Then a class of failures disappears that usually eats up support: answers based on an outdated instruction, a request lost between systems, “drifted” logic after an edit.
A set of regression dialogues from real requests is run on every change, and the previous configuration version is rolled back with one action.
Agent stability is a consequence of observability. Every step is written to a log: which tool was called, what data returned, where escalation triggered. These records show what actually affects answer quality, and editing one branch does not break neighboring ones.
Quality control becomes a regulation, not a one-time check: the sample is assessed by rubrics, discrepancies are labeled and turned into new checks.
The agent logic is described without code in Stack AI — a new version is approved and released in hours, and the effect of implementation is reproducible rather than dependent on a single engineer.
Options for building agents on Stack AI: builder, code, hybrid
The build mode determines not the agent’s “intelligence,” but who can change it after launch and how quickly.
Stack AI offers three approaches: a visual agent builder, custom assembly, and a hybrid, where the dialogue framework is assembled with a mouse and critical nodes are written by an engineer.
The difference shows up in maintenance: a scenario in the builder is edited by the department head, while logic in code is edited only by a developer.
| Option | When it fits | Limitations |
|---|---|---|
| Visual builder | Processes with clear steps: request intake, request routing, answers from the knowledge base | Complex branching and non-standard integrations run into the set of ready-made blocks |
| Custom agent build | Non-standard logic, own models, strict data processing requirements | Edits go through a release cycle; an engineer is needed for support |
| Hybrid | A typical scenario with one or two unique sections: calculations, reconciliation with an accounting system | You need to agree in advance on the boundary: what is edited in the builder and what in code |
The choice of implementation mode comes not from a buzzword, but from three questions: how often the process rules change, how many external systems need to be touched, and how strictly data access is regulated. Frequent edits to wording and thresholds — builder.
Rare but deep changes — code. When a process lives and changes but contains a section that the builder cannot describe, a hybrid is cheaper and more reliable: the boundary is fixed in the documentation, and both sides know where to look for the cause of a failure.
Order of work: from process audit to handover to support
Agent implementation starts not with configuration, but with process analysis: what requests come in, who processes them now, where time is lost, and at which steps a person is needed.
The audit takes several days and immediately shows which sections to give to the agent and which to leave to people. Its results produce a role diagram, responsibility boundaries, and a handover route to operation.
- Process audit. We analyze the flow of requests, calculate the share of repetitive actions, and mark steps where the cost of an error is high. The output is a list of operations for automation and the points where a person remains mandatory.
- Agent and role diagram. We determine what data the agent reads, which systems it accesses, and what it does when information is insufficient. We fix where it answers itself and where it prepares a draft for the operator.
- Scenario setup. We assemble dialogue branches and handover rules to a human in Stack AI, and connect knowledge bases and reference guides. Each scenario has boundaries: topic, permissible actions, stop condition.
- Testing on real data. We run the agent on historical requests, not invented examples. We calculate the share of dialogues closed without an operator and analyze every erroneous answer.
- Edits based on testing results. We refine wording, add missing rules, and remove branches that lead to a dead end. Two or three iterations are usually enough for the agent to stop making things up and correctly hand complex matters to people.
- Handover to operation. We give the customer’s team the agent diagram, scenario descriptions, knowledge base update rules, and rollback procedure. We conduct training so that edits can be made without our involvement.
The route ends not with launch, but with removing dependence on the contractor: the team understands how to change scenarios and where to look when deviations occur. Describe your process — we will return with an estimate and a plan for these stages.
Configurations, regulations, and documentation on output
The project is closed not when the agent answers demo requests, but when your team can change, stop, and restore it itself.
Therefore, along with the working scenario on Stack AI, we hand over a package that an engineer who did not participate in development can understand without us.
- Handover configurations — export of scenarios, agent instructions, tool sets, and routing rules. They are deployed in your environment without “from memory” edits.
- Integration diagram — data sources, connection types, queues, request rate limits, and failure points with instructions on what to do if a connection breaks.
- Access matrix — employee roles and service accounts: who gets read, who gets launch, who gets scenario editing and publication to the production environment.
- Operation regulations — what is checked and how often, which metrics count as a deviation, who responds to errors, and in what order.
- Agent documentation — request processing logic, input and output data, dependencies on external systems, and typical causes of failures.
- Change regulations and rollback — how an edit is tested, who approves publication, and how quickly a return to the previous version is performed.
- Set of reference checks — typical requests with expected answers: after each edit, you can see whether what already worked has broken.
Everything handed over resides in your accounts and your repository — the agent’s operation is not tied to our team.
Case study: request triage in support — metrics before and after
A service support team came to us with a flow of 6–8 thousand requests per month, and the entire queue was triaged manually. An operator opened an email, determined the topic, searched for the order in the accounting system, checked the status, and wrote a reply.
Typical questions — where is the package, how to return a product, how to renew a subscription — consumed more than half of working time, although they required the same actions as complex investigations.
How the agent loop is set up
The loop was assembled on Stack AI, without developing models from scratch: requests from email and the website form enter a single input, the text is normalized, the agent determines the topic and extracts mandatory fields — order number, category, essence of the request.
Then the decision diverges into three branches: a typical one is closed with an answer from the knowledge base after checking the condition, an incomplete one is returned to the client with a clarifying question, and a non-standard one goes to the operator together with a ready draft and the found order.
Disputed topics, claims, and anything mentioning refunds or legal wording are passed to a human immediately — without attempting auto-close. Every step of the agent is logged in the ticketing system, so error analysis follows actual chains, not impressions.
Metrics before and after
| Indicator | Before | After |
|---|---|---|
| Requests closed without an operator | 0% | 46% |
| First response time | 4–6 hours | 3–7 minutes |
| Full processing of a typical request | about a working day | 10–20 minutes |
| Routing errors | 18% | 4% |
The automation result rests not on the model, but on the regulation: operators label disputed investigations, and this labeling goes into the knowledge base and branching rules once every two weeks.
After launch, the share of repeat requests on the same topic fell by about three times — the client receives an answer in the very first message, not after a back-and-forth.
What to do if the agent is already built on another builder
You cannot transfer an agent between builders “as is”: platforms describe branching, tool calls, and context storage differently. But the value here is not in being tied to a platform, but in the content — role texts, behavior rules, the knowledge base, the list of integrations. This transfers and determines how quickly the agent reaches its previous quality level.
System prompts, the document structure of the knowledge base, ready-made answer wording, and the list of external systems with access rights transfer almost unchanged.
What gets rebuilt is what describes behavior: conditions for transitions between steps, the order of tool calls, error and retry handling, and rules for handing the dialogue to a human.
These parts are assembled anew from the logic description — they cannot be transferred by copying blocks.
To avoid losing previous work, we start with an inventory: export the dialogue history and the list of scenarios, and separate working ones from drafts. From live requests, we assemble a control set — several dozen typical and edge-case requests with reference answers.
This is a measurable benchmark: after rebuilding, we run the same set and see where behavior diverged.
The framework is raised on Stack AI and tested on the control set before switching traffic. Then shadow mode: some requests are processed by both agents, answers are compared, and discrepancies are analyzed manually.
Traffic migration is gradual, with the ability to return to the previous platform until the new one shows a comparable result.
Technical doubts before starting implementation
Objections at the technical review are usually the same: how to verify answer quality before operation, what to do with an accounting system without a ready-made connection, how to limit the agent’s rights, and who is responsible for operation after launch. We answer to the point — all checks take place before production operation.
How to make sure answers are high-quality before operation?
We assemble a test set of real employee requests and label reference answers together with your team. The run shows the share of correct and incorrect answers; we analyze errors by type: wrong source, outdated data, incorrect interpretation. We fix the acceptance threshold in writing — if it is not reached, operation does not start.
What to do if there is no ready-made connection for the accounting system?
We connect through the system’s application programming interface, webhooks, or a scheduled export to intermediate storage. On Stack AI, the scenario is assembled on top of these sources without modifying the accounting system itself. We choose the update mode based on the task — event-driven or scheduled, with a discrepancy log.
How to limit the agent’s access to data?
The agent works under a separate service account with read-only rights to the necessary registries. Answer filtering is based on user attributes: an employee of one branch will not receive data from another branch, even if they rephrase the request. All requests are logged — who asked, what they received, and from which source.
Who supports the solution after launch?
We hand over the regulation: how the knowledge base is replenished, who approves changes to answer rules, which metrics we monitor — the share of unsuccessful answers, response time, and the share of handovers to a live specialist.
We run scheduled checks on every change to accounting systems; otherwise, the agent starts answering from outdated reference books.
Let's discuss the task and return with an estimate and work plan
To ensure the estimate of the scope of work is based on facts, not assumptions, at the input we need only a short set of information. Describe the task as it looks inside your organization — this is enough for us to return with an implementation plan and clear stage boundaries.
- scenario goal and acceptance metric: by which indicator you will understand that the agent is working — share of requests closed without an operator, response time, accuracy of request triage;
- composition of source data: where regulations, the knowledge base, and correspondence history are located, who owns these systems, and who grants access;
- 10–20 real examples of requests and reference answers — they show where the solution errs and what will need additional training;
- data and environment constraints: personal data, security service requirements, where the agent can physically operate;
- roles: who on the customer side accepts the result and who participates in stage acceptance;
- what has already been tried and why it did not fit — this saves one or two rounds of iterations.
Then, during the review, we clarify what cannot be read from documents: peak load, behavior when an external system fails, the format of the answer to the user, and the points where a human is absolutely required.
If you are choosing between a ready-made no-code builder like Stack AI and a build on a software integration — we will compare both options against your scenarios, not general descriptions.
We return with an estimate of the scope of work, a stage-by-stage implementation plan, team composition, timelines, and a list of risks together with how we mitigate each one. The project starts after the plan is approved and access to data is opened. Describe the task in free form: this is enough for us to return with an estimate and work plan.
