ChatDev implementation for multi-agent software development

ChatDev in production: where the multi-agent pipeline falls apart

In a demo, multi-agent development looks convincing: a small isolated task, a clean repository, a predictable result in one run.

In a working repository, the pipeline encounters foreign code, a long change history and parallel branches — and here failure points appear that simply were not in the demo.

The first and most frequent is role desynchronization during agent orchestration. The architect describes the solution in one set of files, the implementer edits others, the reviewer looks at a third: formally all roles worked, but the output is an inconsistent set of changes.

While there is no explicit contract between roles (who reads which paths and changes which), the pipeline gives a false sense of completion.

The second is unlabeled edits. The agent goes beyond the declared zone and changes configuration, tests or dependencies without marking it. In the common set of changes, this is mixed with the target, and the reviewer spends time on archaeology instead of logic. This is closed by attribution: each edit is tied to a role, step and run number.

The third is a non-reproducible run. If input data, model version, assembled context and tool call results are not fixed, a re-run yields different code, and rollback and incident analysis become impossible. Below we analyze how this is closed at the configuration and process level.

What a managed agent development pipeline provides

Multi-agent development gives a measurable effect only when agent roles, boundaries of their rights and mandatory human review points are defined in advance.

We deploy ChatDev as a role orchestrator — architect, developer, reviewer, tester — and fix for each class of tasks a route: what the agent does itself, where it stops and waits for an engineer.

The first metric is the share of autonomous tasks reaching merge without manual rework. On typical changes, such as logic edits, test coverage or configuration updates, it reaches stable values, and review stops going to mechanical edits. Revision cycle time shrinks because checks run in parallel with code writing, not after it.

The second is an eliminated class of failures. Changes that fail automated code checks, static analysis and tests do not reach the main branch: the reviewer agent returns them to the cycle and does not create a merge request until a green run. This removes a layer of incidents that were previously caught in review or already in production.

Traceability of agent decisions is what, without it, turns autonomy into risk. Every step is written to a log: which role, based on which files and check results, made a decision, which artifact it produced and who confirmed it. From the log, the logic of the change is restored and it is visible where a human intervened.

Next we will analyze how these metrics are calculated, by what thresholds the decision to expand autonomy is made, and what remains with the engineer at each stage.

Run modes: interactive, batch, autonomous

Run mode determines where the human stops in the pipeline and what can be accepted by formal criteria. In ChatDev this is a task-level switch: the same role chain either goes with confirmation at each step, or passes the cycle entirely. Below are three modes and what restrictions each imposes on control and acceptance.

In practice, modes do not exclude each other. Batch is the default working mode: predictable run time and verifiable acceptance.

We enable autonomous in areas where there are tests, test data and fast rollback, and we leave interactive for first passes and tasks where requirements are still being clarified.

Implementing ChatDev: from repository audit to handover

Implementation of ChatDev starts not with launching agents, but with an inventory of what already exists in the code. The order of stages is the same for any repository: first we capture facts, then design roles, and only then give agents access to work. Each stage ends with an artifact that can be checked — without it, the next step does not start.

1. Repository audit. We look at module structure, how the project is built, where tests are and which critical paths they cover. The output is a dependency map and baseline metrics: build time, share of paths under checks.
2. Agent role scheme. We define who designs, who writes code, who reviews and who runs tests, and at which steps the task moves on. Agents do not write to protected branches directly.
3. Environment setup. We deploy isolated environments with read access to the main branch, and keep keys and tokens outside the code. We set limits on context size and number of iterations — the run does not go into an infinite loop.
4. Dry run. We take one small task from the current queue and run it end-to-end, from statement to changes. We look at how many edits were needed after the agents and where roles handed off work with losses.
5. Acceptance criteria. Before production launches, we fix: tests pass, static analysis gives no new findings, each change has a rationale and trace. A report for each run is built on these criteria.
6. Handover to the team. We provide run regulations, task templates and analysis of typical failures. First runs go with our support, then the team runs the cycle itself.

Configurations, role schemes and run regulations

After the first run in the customer's repository, what remains is not a set of demo scenarios but a working kit: it is enough to launch multi-agent development again without us.

All artifacts are versioned together with the code — editing a role or access rule goes through the same acceptance review as a change in a service.

• Role map — which roles are allocated for the task, which part of the work each is responsible for and by what criterion its result is accepted.
• Agent access rules — what a role reads, where it writes, where a human check is mandatory before merge.
• Run configurations — team composition, task handoff order, iteration limits and stop conditions, so the result does not depend on the executor.
• Rollback regulation — which save points are made before start, how to return to the previous state and what we consider a signal for rollback.
• Edit conflict resolution — arbiter for a disputed edit, mandatory checks and the order by which two roles are brought to one decision.
• Run log — where input data, steps and result are stored: from it a failure is reproduced, not discussed from memory.
• Check catalog — tests and static analysis that a run must pass before merge to the main branch.

The kit is transferred to your repository, and then the team runs it itself: changes role composition, adds checks, creates new configurations for changed tasks. We configure ChatDev so these files are read and edited like ordinary project code.

Case: reducing edit cycle on an agent stand

Before implementing the stand, the edit cycle in an internal service took 9–11 days: a task went through review, returned for rework and queued again.

About a third of tasks reached merge to the main branch — the rest were handled manually, and contract desynchronization between modules was caught already at integration, when the cost of a fix is highest.

What changed

We assembled an agent stand on ChatDev: the roles of analyst, developer, reviewer and tester work in one dialogue and pass artifacts to each other — a refined statement, a patch, a set of checks.

Each task lives in a separate branch, and a closed list of checks lets it into main: build, tests, signature and migration verification.

What resulted

The share of tasks reaching merge on the first pass grew from 35% to 78%, and median edit cycle time decreased from 9–11 days to 2–3.

The class of failures "modules diverged on contract" stopped reaching the main branch: the reviewer agent verifies signatures before build, and the discrepancy surfaces at the statement stage, not integration.

The engineer's role remained decisive: final merge and disputed cases are with a human. If the agent does not converge on a decision, the task returns with a full dialogue log and a list of rejected options — analysis takes minutes instead of hours.

How ChatDev differs from manual assembly of agent scenarios

The difference is not that one approach is smarter, but where responsibility for the run result lies. In a self-written agent bundle, roles, call order and context transfer format live in code written for a specific scenario.

While there are five tasks and all of the same type, it is fast and flexible. When there are dozens of scenarios and input data is heterogeneous, the bundle starts behaving differently on identical requests.

Reproducibility of runs suffers first: state is smeared between process memory, intermediate files and prompts that were edited along the way. Repeating a launch is possible only if inputs, role versions and step order were saved from the very beginning.

Failure diagnostics for the same reason boils down to reading code and rerunning — the log shows what failed, but not what the input was.

As load grows, a self-written bundle becomes more expensive not in development, but precisely in pipeline maintenance: each new request type pulls edits into shared code and re-validation of already working roles.

A ready pipeline covers step recording, reruns and routing — refinements remain point, not end-to-end.

Tech team doubts: control, rollbacks, isolation

We limit agent rights at the access token level: ChatDev receives credentials only for reading the repository and writing to a separate service branch.

Merge is done by a human via manual check, secrets do not get into the agent context, and container network access is limited to an allowlist of addresses. The rights manifest is checked on every launch — going beyond scope stops the run, not corrupts the code.

Looping is the second common story: agents rewrite the same file in a circle. We set a hard limit of iterations and steps per task, monitor whether the final divergence from the base version changes, and when changes stop, the run ends with a report on reasons. Plus a ceiling on context size — a long cycle does not turn into uncontrolled consumption.

We embed the pipeline into a legacy repository gradually: one module, isolated branch, characterization checks on current behavior. The agent result goes through the same builds and checks as manual edits — public contracts do not change silently.

What if legacy code has no tests?

First we cover critical paths with checks that capture current behavior, and only then allow the agent to make edits. Any discrepancy is visible at build, not after release in the working environment.

Let's discuss the task: what's needed to estimate implementation

To return with an estimate and work plan, rather than general wording, we need inputs on the repository, team and current processes.

Multi-agent development is embedded into the existing environment, so first we look at how build, checks and change acceptance are arranged — both the scope of work and which areas can be given to agents immediately and which left to people depend on this.

• Repository structure: languages, number of modules, state of build and tests, where the main logic is concentrated.
• Branching and version release: how changes are prepared, who confirms merge, how often releases happen.
• Automated checks: what runs on each change, where checks are still done manually.
• Team composition: how many engineers own the code, who accepts the agents' result and how mutual change review is arranged.
• Task workflows: how requirements are formulated, where tracking is kept, by what criteria a task is considered done.
• Environment constraints: isolated run environments, access, rules for handling code and data.

This set is enough to return with an estimate and work plan: which areas we cover with agents, what remains with engineers, how we check the result at each step. If some inputs are not collected yet — no problem, we will clarify in conversation and suggest how to get them. Describe the task — we will return with an estimate and plan.