Implementing Factory AI for autonomous software development: experience and results

Why autonomous development with Factory AI breaks at scale

On a single task, autonomous development with Factory AI looks convincing: the agent receives a specification, finds the necessary files itself, makes edits, and runs checks.

While tasks come one at a time, the bottleneck shifts to the specification — the more precisely the outcome is formulated, the fewer iterations are needed before an accepted change.

In a stream, the picture is different. Several agents concurrently touch shared modules: data schemas, shared utilities, build configuration.

Each sees the repository in its own version, and change conflicts surface not at the moment of editing but at merge — when a solution that has already been verified and accepted has to be rewritten. It is at this step that autonomous development with Factory AI stops scaling linearly.

Then a second effect comes into play — false check failures. The test suite is designed for sequential integration, and with parallel branches failures come from other people's edits.

The agent reads a red run as its own error and fixes things unrelated to the task: the scope of the change grows, and review of edits loses predictability.

The task queue does not dissolve; it grows: each task accumulates rebases, repeated runs, and review of other people's failures. The time to complete a single task increases nonlinearly, and the gain from automation is consumed by coordination.

We examine how this is addressed — with queue rules, branch isolation, and boundaries within which the agent does not go beyond its task.

Measurable effects: what changes after agents are introduced

A team's throughput is measured not by agent activity but by the number of changes that reach the main branch per cycle.

After agents are introduced, this counter grows because routine edits stop competing with tasks that require engineering experience: dependency updates and typical changes move into the background and no longer occupy the entire review.

The share of autonomous tasks is the second metric: how many changes the agent carries from edit to check run without intermediate human intervention.

Factory AI covers recurring scenarios — dependency updates, template-based migrations, same-type configuration edits — while the engineer gets involved at the review and risk assessment stage, not at every step.

Recovery time is reduced due to ready context: the agent has access to edit history, build logs, and check results, so a draft fix appears before an engineer would manually reconstruct the failure picture.

Rollback remains manageable — changes arrive in small batches that can be removed individually without rolling back the entire release.

A separate consequence is that a class of failures related to human oversight goes away: a forgotten migration, configuration desynchronization between environments, an unclosed branch.

Such failures are costly not because of complexity but because of frequency, and the agent eliminates them by checking against a checklist before the change moves further through the environment.

Agent operating modes: comparison of approaches and limitations

An agent's operating mode is primarily a mode of writing to the repository, and it defines the class of risks, not a "level of autonomy." In Factory AI, switching is done at the task level: the same model in suggestion mode and in automatic mode produces a different result in terms of control over changes. Below are three alternatives that teams actually choose between.

The common limitation of automation is the same: the agent sees code and tests but not agreements, reasons for past decisions, and data requirements.

Therefore, in any mode there remains a point where human review is needed — acceptance of changes at module boundaries, access rights, handling of personal data.

The mode is selected for the task, not for the service: for routine edits, an isolated copy is sufficient; for the product core, mandatory review is required.

Implementation stages: from task audit to handover of regulations

Autonomous development produces an effect not by itself but as part of a process with clear boundaries: where the agent works on its own, where a human is needed, who is responsible for rollback.

Therefore, Factory AI implementation proceeds in stages — from task audit to handover of regulations — and each stage has a verifiable result.

1. Task and rule audit. We analyze which work repeats — typical edits, tests, dependency updates — and which rules the team already follows: style, review, coverage requirements. The output is a list of tasks for the environment and a list of what it does not touch.
2. Environment design. We determine where agents run, how they gain access to the repository, and how the result reaches the main branch. An isolated environment per task is a baseline requirement.
3. Role and access setup. We delineate rights: read, write, where an approving human is required. Keys live in a secret store and are issued for the minimum term.
4. Pilot on a narrow area. One service or one class of tasks where the result is easy to measure. We look at the share of accepted edits without rework and the time from task statement to verified change.
5. Quality check. We compare against the team's expectations: whether changes pass mandatory checks, whether the share of rollbacks is growing, whether reviewers find readability acceptable. If performance drops, we adjust rules and prompts rather than expanding scope.
6. Handover of regulations. After a stable pilot, we connect the remaining teams and hand over documentation: the environment design, access matrix, review rules, rollback procedure, and analysis of typical failures. Engineers run the environment themselves.

Deliverable package: configurations, access policies, documentation

Implementation does not end with launch: the customer's team retains a working package that can be used to develop the system and hand it over to other engineers without us.

We capture in it what usually lives in chat threads — which configuration files are responsible for what, where the boundaries of rights lie, and at what point a human must be brought into the process.

• Configurations — environment, agent launch rules, and repository connection settings, with an explanation of what changes and what it leads to.
• Integration design — how Factory AI is connected to repositories, branches, and the task queue: where a task is created, where the result appears, where confirmation is required.
• Access policies — a role-based rights matrix: what the agent reads, where it writes, which operations it performs only after engineer approval.
• Agent roles — the purpose of each role (development, verification, failure analysis) and its boundaries in a disputed situation.
• Verification rules — change acceptance conditions: mandatory checks, thresholds, reasons for automatic rejection.
• On-call regulations — actions of the on-call engineer when the task queue stops, the rollback procedure for an unsuccessful change, and the escalation point.
• Documentation — launch instructions, typical failures and how to analyze them, and a log of settings changes.

The package remains in your repository and is updated together with the system — this is what makes knowledge transfer reproducible rather than dependent on a specific engineer.

Factory AI implementation case study: metrics before and after

We implemented Factory AI in a live business area where the task queue held at several dozen: API edits, data schema migrations, interface updates.

We measured three metrics — task lead time from statement to merge, the share of edits closed autonomously, and the share of returns for rework. Measurements ran for two weeks before implementation and the same after — so that one-off spikes would not blur the comparison.

How it was before

The task went through the chain manually: the developer reads the specification, finds related code, writes the edit, runs tests, responds to review comments.

For a typical change, this took 2-3 days, and writing itself accounted for a smaller part — the rest went to context analysis and switching between tasks.

About every fifth task was returned for rework: an adjacent case was not covered, contracts diverged, an extra module was affected.

What we changed and what the result was

The preparatory part was given to agents: gathering context across the repository, drafting the edit, and running tests before human review.

The engineer approved the plan and accepted the result; the merge rule was unchanged — nothing goes into the main branch without green checks.

Task lead time decreased to 6-9 hours, the share of edits closed autonomously grew to four out of ten, and returns for rework fell from 21% to 7%.

The effect holds for typical changes: tasks that alter architecture remain entirely with the engineer — there, the agent speeds up code reconnaissance but not the decision.

How an autonomous agent differs from code autocompletion

Code autocompletion works at the line level: the model sees the open file and neighboring fragments, suggesting the next expression as you type. It does not know what task you are solving and does not check whether the project will build after the suggestion. Accepting or rejecting it is the engineer's decision at every step.

An autonomous agent receives a task as a formulation — a defect description, behavior requirements — and expands it into an edit plan. It determines which modules to touch, in what order to make changes, where new tests are needed, and what the edit will entail.

Edit planning happens before the first file change, so the output is a coherent set of changes rather than a scattering of suggestions.

The key difference is verification rules. The agent cannot consider the work complete until the tests, static analysis, and type checks designated as mandatory pass. A failed check returns output into the loop: the agent reads the error, fixes the code, and runs the check again.

The number of iterations is limited, and when attempts are exhausted, the task is returned to a human with a report rather than a silent "seems to work."

In our practice, we connected Factory AI at exactly this level: the agent worked in a separate branch, with a prohibition on touching build configurations and data migrations, while merging remained with the reviewer.

Hence the difference in purpose: autocompletion saves typing time, while an autonomous agent shortens the path from task to verified change — provided that verification rules and the boundaries of its work are described in advance.

Technical concerns: limitations, risks, control

Autonomous development does not remove engineering limitations — it moves them into the area of configuration and control.

Below are five questions asked before starting Factory AI implementation: access to secrets, environment isolation, quality measurement, the fate of unclosed tasks, and compatibility with rules already deployed in your repository.

Do edits go directly into the working branch?

Work happens on an isolated copy — a separate environment, its own test data, no calls to production databases. Changes reach the main branch only through code review and the mandatory automated checks already configured in the project.

How do you measure the quality of the result?

Three metrics: the share of paths covered by automated checks; the share of changes accepted without human edits; time from defect detection to rollback. They are captured at every stage, so degradation is visible before release, not after.

What happens to tasks the agent did not close?

The work does not disappear: a branch with history, a draft of changes, and an attempt log remain. The task is marked and handed to an engineer together with the context — they continue from the same point rather than reconstructing the reasoning from scratch.

Is this compatible with our rules and linters?

Yes. Linter settings, commit templates, branching schemes, and already deployed rules are read from the repository as is. The agent adapts to the existing way of working rather than requiring the configuration to be rewritten for it.

Let's discuss the task: we'll analyze your environment and come back with a plan

So that timeline estimates are based on facts rather than market averages, we start with input data about your development environment.

A full audit is not needed — it is enough to describe the current state, and we will come back with a Factory AI implementation plan broken down by stages.

• How repositories are structured: languages, build systems, share of legacy code.
• Review and release processes: who approves changes and which checks are mandatory before deployment.
• Task tracking system and requirements format — they determine how autonomously the agent can work.
• Test environment: how close it is to production and which checks run automatically.
• Data restrictions: what is prohibited from leaving the perimeter, whether isolation inside your infrastructure is required.
• Acceptance criteria and metrics: share of tasks without edits, time to release, check coverage.

In response, you receive an implementation plan: stages, the order of connection to your environment, access requirements, the composition of the pilot task group, and timelines for each stage with result verification points.

To discuss the task, a short description of the environment and your expectations is enough. Describe it — we will come back with a timeline estimate and an implementation plan.