Developing an AI agent orchestrator: where the process breaks
An agent chain breaks not at response generation, but at the junctions between steps.
One agent passes the result to another in free form, the second asks again and launches the same tool anew — this is how an agent chain failure turns into duplicate calls: balance debited twice, a request created twice, an email sent twice. The log shows the result of a step, but not why it was made at all.
The second failure point is loss of task state. A dropped connection, a service restart, or a timeout in the middle of a scenario zeroes out the context, and the agent goes from the first step while already having partially completed actions behind it.
This is closed not by retries, but by a step log and checkpoints: every transition is recorded, and a rerun continues from the last confirmed point.
A separate problem is access boundaries. An agent with broad permissions accesses internal systems directly: writes to the database, calls administrative methods, pulls data beyond the scope of the task.
Restrictions must be at the level of tools and roles, not in a text instruction — wording in a description can be bypassed, an access policy cannot.
And above all is the opacity of the agent's decisions. An answer arrives, but there is no way to answer the questions of which data was used, which tools were called, and where the logic went wrong. We break down how each of these areas is closed.
What changes after implementing managed agent systems
The effect of an agent system is measured twice — before implementation and after a month of operation: scenario completion time, share of requests without manual intervention, number of failures per thousand operations, and the class of failures that stopped occurring.
Below is the checklist by which we accept the work: each item is checked on a live stream of tasks, not on a demo scenario.
- Scenario execution time. A chain of 5–8 steps runs without pauses for handoffs between people; approval waiting remains only where required by policy.
- Share of requests without an operator. Calculated separately for each request type: routine ones go into autonomous mode, and a human gets involved for exceptions and non-standard cases.
- Reduction of manual operations. Data reconciliation, exports, document preparation, and task creation are moved into agent steps with a fixed result, not into instructions for an employee.
- Failure resilience. Retries, timeouts, and a hard limit on the number of steps remove the failure classes "loop with no exit" and "context loss" — they are visible in metrics rather than surfacing for the client.
- Observability. Every agent step leaves a trace: input, invoked tool, response, duration. Tracing is built on the open standard OpenTelemetry, so data is not locked into one vendor's format.
- Recovery. Rolling back to the previous orchestrator configuration takes minutes and does not require rebuilding the connections between agents.
- Resource consumption. Simple requests are routed to lightweight models, heavy ones to powerful models; load stops growing linearly with the number of requests.
These seven points provide one picture: how much work the system handles on its own, where it breaks, and how quickly it returns to service.
Options for building an agent system: comparing modes
The operating mode of agents is not about "smarter or simpler," but about where you are willing to pay with complexity: at the input, at runtime, or in debugging.
The same scenario can be assembled in five ways, and the choice is determined by the cost of an error, the length of the step chain, and how predictable the order of actions is.
| Option | When it fits | Limitations |
|---|---|---|
| Single agent | Short scenarios with 2–4 tools, user dialogue, hypothesis testing | Call order is not controlled; context is lost on a long chain, and failure is hard to localize |
| Rigid step graph | Regulated processes with a fixed order: request intake, document reconciliation, routing | Deviation from the scenario requires editing the graph; handles branching and incomplete data poorly |
| Control layer with a task planner | Tasks with a variable set of steps: data collection from multiple sources, analysis, summary | The planner must be limited by number of steps and depth, otherwise loops appear and response time grows |
| Event-driven scheme with queues and retries | Long-running and external operations, integrations where node failure is a normal situation | Tracing and debugging are harder; duplicate control and handler idempotency are needed |
| Hybrid: graph plus planner on selected nodes | Production processes where critical steps are regulated and the variable part depends on data | Requires an explicit state machine and escalation rules, more expensive to design and maintain |
In practice, working systems most often converge on a hybrid: a rigid graph holds critical steps and human approval points, a planner covers the variable part, and queues and retries are responsible for delivering the result when external services fail.
Such a scheme allows a phased transition — graph nodes call the same tools as an autonomous agent, so the mode changes without rewriting the logic from scratch.
Implementation stages: from process audit to handover
An orchestrator is not a single agent, but a loop of several roles, tools, and rules by which they exchange results. That is why the work proceeds in stages: first we capture the processes, then we build the scheme, then we test it against failures.
- Process audit — we analyze which operations repeat, where decisions are made by rules, and where a model is needed. The output is a scenario map, a list of data sources, and points where a human must confirm an action.
- Designing the loop scheme — we determine the composition of agents, their roles, the order of task handoff, the state storage location, and the boundaries of authority. Here we also describe the exchange rules: message format, maximum response waiting time, retries on failure.
- Setup and assembly — we connect sources and tools, configure task routing, limits, logging, and tracing so that every agent step can be reconstructed.
- Failure test scenarios — we check behavior with an unavailable source, an empty response, and a conflict between two agents. The loop must not go into an infinite loop or perform the same action twice.
- Pilot run on one scenario — we roll the process out to a limited group, compare the result with manual execution, and adjust the exchange rules based on actual results.
- Handover and team training — we hand over schemes, policies, accesses, and logs, conduct training, and remain on support until the team starts changing scenarios on its own.
After that, new roles and logic branches are added by the customer's team — without another engagement from our side.
Policies, schemes, and configurations: what remains with you
The development of an AI agent orchestrator ends not with a demonstration of a working scenario, but with the handover of a package that your team can use to maintain the solution without us.
We hand over not archives and screenshots, but working artifacts: what is in the repository, what is applied in the loop, and what describes the system's behavior during a failure.
- Orchestrator configurations: descriptions of routes between agents, executor selection rules, parallel run limits, timeouts, and retry conditions — in version control, not only in a database on the server.
- Loop scheme: what is deployed on your side, where states and logs are stored, which external services are involved, and what will happen if each of them is unavailable.
- Access policies: roles of operators and services, which data and tools an agent is allowed to access, where keys are stored, and how they are rotated.
- Scenario descriptions: which tasks go to the agent automatically, where a human is mandatory, what counts as successful completion, and which results are unacceptable.
- Incident investigation procedure: where to view the run log of a specific task, how to roll back a problematic scenario, and who decides to stop it.
- Change procedure: how instructions and routes are edited, which checks are run before deployment, and how rollback to the previous version is prepared and tested.
- Maintenance documentation: instructions for local launch and debugging, a description of extension points, and a log of decisions made with rationale.
The scope of the package depends on the number of scenarios and integrations — let's discuss the task, and we will come back with an estimate and a handover plan.
Case study: how a managed loop removed manual steps
Before implementation, the flow of client requests relied on an operator: they manually parsed the request, determined its type, pulled data from the accounting system, and decided which specialist to pass it to.
A typical scenario took 18–20 minutes; at peak shift times, the queue grew faster than it was processed, and some requests went out with an incorrect category — they were returned for reprocessing.
We started not with the model, but with a process map: we wrote out the steps where a human makes the decision and separated those where there is no choice — only rule-based reconciliation. These turned out to be the majority.
The loop was assembled from three roles: a classifier agent, an executor agent, and a validation layer that passes the result onward only after schema validation.
Orchestration is built on LangGraph: task state is stored explicitly, so a failed step is replayed from the point of failure, not from the beginning.
What exactly was changed
The main thing is not the autonomy of the agents, but the boundaries. Each has a limited list of allowed actions, a step limit, and a mandatory escalation point to an operator when the model's confidence is low.
Requests where the loop is uncertain go to a human with the context already assembled: the operator does not search for the data again, but confirms the decision.
After launch, processing time decreased from 18–20 to 4–6 minutes. The share of requests closed without manual intervention reached 78%; the remaining 22% are not failures, but deliberate escalation where the cost of an error is higher than speed.
Failures that previously led to reprocessing — loss of context between steps, incorrect category, duplicate record in the accounting system — dropped from 34 to 3 per thousand operations.
This class of failures is closed architecturally: validation rules live in the loop itself, not in an instruction for the operator.
How does an orchestrator differ from a set of scripts and queues?
A separate script solves one task: it starts on an event or schedule, runs, and records the result. While there are three steps and the failure is visible to the eye, this is enough.
But as soon as a task requires branching, external services, and human confirmation, the script falls apart: no one stores at which step everything stopped and what has already managed to change.
| Option | What it can do | Where it breaks |
|---|---|---|
| Set of separate scripts | Linear launch on an event or schedule | No shared task state: after a failure, it is unclear what was done and what was not |
| Task queue with handlers | Distributes load, retries failed jobs | The handler does not see the context of the whole task; step order relies on agreements between teams |
| Control layer | Plans steps, stores task state, replays failed sections | Requires discipline: step descriptions, idempotency, observability |
The control layer differs in that it holds the task state as a whole: which step has been completed, what data has been received, what we are waiting for from an external system.
Step planning becomes data, not the order of calls in code — the route can be changed without rewriting the processing logic.
Retries also stop being a general queue setting. The orchestrator knows that a debit cannot be retried, while a request to an external service can be, with increasing delay and a retry limit.
A failed section is replayed together with the already accumulated context, not from scratch: a long chain of ten steps does not start over because of a single network failure.
The price of such a loop is its own observability. A transition log is needed, manual investigation of stuck tasks, and a limit on parallel runs; otherwise the control layer itself becomes a source of silent hangs.
Technical questions: reliability, access, control
Concerns about an orchestrator relate not to the model, but to behavior at the boundaries: external system failure, agent permission boundaries, a failure once in a hundred runs. We address what is most often asked before starting.
What happens if an external system does not respond?
An external call is a separate step with its own timeout and retry policy. If the system does not respond, the scenario goes into a compensation branch or pauses rather than returning a partial result to the user.
Every attempt with a response code is written to the log — you can see at which step the chain broke.
How are agent accesses to data restricted?
The agent receives not database access, but a set of allowed actions with a limited set of fields. Permissions are granted for a specific role and live for the duration of the task, while writes and deletions are moved into separate confirmable steps.
How do you debug a scenario that does not always fail?
Runs are traced: a trace in OpenTelemetry links steps, inputs, and tool responses. An unstable scenario is reproduced on a saved input dataset, and disputed cases become regression checks before deploying changes.
Will this integrate into our existing steps, and what limits are there?
Existing steps are connected as separate tools: they do not need to be rewritten, just describe the inputs, outputs, and error conditions.
Request limits are set at the scenario and external system level — the queue and parallel task limit prevent one scenario from exhausting the shared quota.
We address these questions during the design phase — otherwise the first serious external system failure becomes the first resilience test.Let's discuss the task and come back with an estimate and plan
For the estimate to rely on facts rather than assumptions, what already exists in your infrastructure is enough to start: a list of scenarios the orchestrator must cover, a list of systems the agents access, and constraints on access and data.
If you have preserved examples of requests where the logic fell apart, they will save hours of clarifying conversations.
Based on the discussion, we come back not with a verbal description, but with an engineering implementation plan:
- scope of work by stages — from the request routing scheme between agents to the wrapper around external system calls;
- loop scheme: where agents run, how dialogue state is stored, and what limits access to sensitive data;
- milestones and stage timelines with acceptance criteria for each: which set of scenarios must pass without manual intervention;
- list of risks with assessment: quality degradation under growing load, conflicts of parallel calls, incomplete responses from external services;
- requirements for your side — access to systems, test data, the person responsible for acceptance, and the person who confirms business rules.
We suggest discussing the task in detail: we will break it into parts, mark unclear areas, and send an implementation plan with scope of work, timelines, and responsibility areas. Describe the task in any convenient form — we will come back with an estimate and plan.
