Why a browser AI agent breaks on real interfaces
A recorded scenario relies on rigid paths to elements: clicking the third form field, XPath to a button, an identifier that the markup regenerates on every build. As long as the interface has not changed, the browser AI agent completes the route in minutes.
If blocks are rearranged, a banner is added above the form, or a button is renamed, the step cannot find its target, the scenario breaks off midway, and the queue of requests stalls until manual review.
A separate class is interfaces with no API, where the only channel is the page itself: an internal system, a supplier portal, a bank portal.
Extraction goes only through markup, and that markup is dynamic — asynchronous loading, lazy rendering, modal windows over fields, different branches for different roles.
A fixed pause does not help here: on a fast response the robot clicks an empty spot, on a slow one it moves on without waiting for the data.
Failures during data entry almost never look like an error. The field was redrawn after autocomplete, the mask dropped some characters, an invisible layer intercepted the click — the value was not recorded, but the step is formally complete.
The robot moves on and leaves a request with an empty required field or an amount from the wrong row; a human catches this when reconciling with source documents.
This is also why robot maintenance exists: after every interface change, an engineer goes through the steps again and fixes selectors. Fragile automation scenarios take time not for process development but for patching. Below we discuss how this is addressed.
What adaptive interface management provides instead of scripts
An automation script relies on coordinates and a rigid sequence of actions: the button shifts, the field order changes — the scenario fails. The agent reads the interface just like a human: it parses markup, labels, and visual cues. Below is what this changes in operation.
Resilience to interface changes
Adaptive interface management relies on the meaning of an element, not its coordinates. A renamed block, a new banner, a changed field order do not break the scenario, so a package of website-side changes no longer turns into a separate automation rewrite project.
Reduced processing time
Computer vision in automation makes it possible to react to an element appearing rather than waiting out a fixed pause "just in case." In high-volume scenarios, this cuts the time to go through the chain by 2–3 times and eliminates downtime at steps where data loads longer than usual.
A class of failures eliminated
A whole layer of incidents disappears: "element not found," "a window covered the button," "the field order changed." The agent recognizes a pop-up on its own, waits for loading, and selects the needed element by meaning.
Interfaces built with WCAG in mind are navigated more reliably by the agent — element roles and labels give it reliable anchors.
Recovery after a step failure
The agent keeps a step log and screenshots, so a failure in the middle of the chain does not require a run from the beginning: execution returns to the last successful point.
Incident analysis takes minutes — the screen state at the moment of failure is visible, and maintenance comes down to editing the step description rather than rewriting the logic.
Operating modes: scripted, autonomous traversal, and hybrid
The operating mode answers the question of who makes the decision at each step: a pre-recorded route or the model at runtime. This determines resilience to interface changes, the amount of manual rework, and how exception handling is designed.
Classic RPA robots can do only the first method, so a markup change breaks the entire run. Below are three modes that are actually chosen between at project start.
| Mode | When it fits | Limitations |
|---|---|---|
| Recorded scenario | Repeatable operations with the same sequence of steps, a stable interface, requirements for predictability and analysis of every action | Breaks when markup or step order changes; deviation is closed manually and requires editing the scenario |
| Autonomous step traversal | Many sources, the full route to data is not known in advance, markup differs from section to section | The run is longer and more computationally expensive, the result requires verification; not suitable for irreversible actions |
| Hybrid scheme | A long process with a stable core and non-standard sections: variable forms, incomplete data | More complex design and debugging: you need to decide where control passes from the scenario to the model and how each decision is recorded |
We lock in the choice of operating mode before development. If it is postponed, exception handling is designed after the fact — and the agent stops at the first non-standard situation instead of returning with a clear report of exactly where the process deviated.
Implementation stages: from process analysis to handover
The agent is not launched immediately on production systems — first we analyze where manual work hits a ceiling. This removes the main implementation risk: automating a step that was not a bottleneck.
- Process audit. We analyze which operations an employee repeats, at what frequency and volume, where delays and errors accumulate. The output is a list of steps ranked by the gain from automation.
- Step map. We record transitions between systems, branching points, confirmation points, and data sources. The map shows where the agent needs write permissions and where a copy is sufficient.
- Agent setup. We build the scenario: screen recognition, input rules, entry checks. We connect write permissions only after the agent matches the manual result.
- Test environment. A copy of the interfaces and depersonalized data: a run with step-by-step reconciliation against the employee's work. We fix errors here, not in the production environment.
- Exception handling. We separately go through cases where the agent is unsure: a non-standard document, a changed field, a connection loss. It stops, flags the case, and hands it to a human.
- Handover to support. We provide regulations, run logs, control points, and the procedure for updating the scenario when interfaces change. For the first period, we manage it together with your team.
The stages do not go strictly one after another: exception handling begins already at the test environment, and regulations are written in parallel with setup. One thing remains unchanged — reconciliation with the manual result before going into the production environment.
Deliverables: diagrams, access roles, regulations
The delivery package is not a checkbox report but a set of artifacts your team can use to operate the agent and change scenarios without us. Everything described has been verified on the test bench: the documentation matches how the system behaves in the production environment.
- Integration diagram. Which systems the agent reads and writes to, through which accounts, where the failure points are, and how they are bypassed without manual intervention.
- Access policies. A role matrix: who runs the scenario, who sees the log, who edits the rules. The principle of least privilege applies at every step — the agent does not receive more than is needed for the task.
- Failure handling regulations. What to do when the target system is unavailable, page markup changes, or the model's response is ambiguous. It is separately recorded in which cases the agent stops and hands the task to a human.
- Action log. Every agent decision with its rationale, a state snapshot, and a link to the source. The log can be exported, filtered, and analyzed for a specific incident.
- Team instructions. Startup and scheduled shutdown, scenario updates, standard checks after changes on the side of external services.
- Deployment map. Where the components live, how the test environment is brought up, and how it differs from production.
- Change procedure. How a new version of the scenario is validated on the test bench and how the previous one is rolled back if metrics decline.
We hand all this over with a walkthrough: we go through each document together with your engineers, answer questions, and assign owners. After that, the team runs the agent itself — or we stay on support.
Case study: requests in the customer portal without manual operations
Processing requests in the customer portal is about 1,800 inquiries per week. An operator opened the card, reconciled the data with the attachment, transferred fields into the accounting system, changed the status, and sent a response.
Eight to nine actions per request, average time around six minutes, and any missed checkbox came back as a repeat email.
What broke in the first approach
The first version was built on scripts tied to page elements: the robot found a field by selector and inserted the value. The scheme held until the first interface update — after the portal moved to Bitrix24, the markup changed, and half the steps lost their footing.
Runs failed on forms with dynamic loading, and the script could not read scans of attachments with different layouts at all.
How it was redone and what changed
The agent with computer vision works not with markup but with what is visible on the screen: it finds a field by label and position, and the language model decides which value fits there. The attachment is recognized and reconciled with the card fields; disputed cases go to an operator.
The share of manual operations fell from 100% to 12%, average processing time from six minutes to forty seconds, and input errors from 3% to 0.4%. Updating the customer portal no longer stops the flow: the scenario searches for elements anew on every run.
How a browser AI agent differs from classic RPA
Classic RPA is a recorded scenario: button selector, click coordinates, fixed step order. As long as the interface does not change, it works quickly and predictably. But rename a field or change the markup, and the step fails, and the process stalls until manual correction.
The browser AI agent does not store a sequence of actions. It receives a goal — "collect items from the customer portal into a table" — looks at the page both as an image and as a structure of elements, and decides on its own where to click at each step. A markup change is not a failure for it but a new observation.
| Criterion | Classic RPA | Browser AI agent |
|---|---|---|
| Element search | Selector, click coordinates | Visual and semantic cues |
| Markup change | Step fails, correction needed | Adapts the method of action |
| Process setup | Faster with a stable interface | Longer to reach a stable result |
| Maintenance | Correction after every release | Refinement of rules and checks |
The difference in labor costs is visible not at launch but over the long run. A scenario on Selenium is assembled faster, but every interface update puts it back into rework.
The agent is more expensive to debug, but maintenance shifts toward refining rules, and this volume does not grow with every release of the external portal.
The choice depends on the environment. An internal system with predictable releases remains the domain of classic RPA. The agent navigates external portals with frequent updates and dynamic windows more reliably.
The working option is a hybrid: critical steps are deterministic, unstable sections are given to the agent.
Technical concerns: access, security, fault tolerance
Engineering teams usually raise the same objections: what access the agent gets, what it does in a closed environment, and how it behaves when the scenario breaks. Below are short answers to each point.
What permissions does the agent need?
A separate service account with a minimal set of permissions — only the sections and operations described in the scenario. Expanding permissions goes through standard approval, and access is revoked with a single operation.
Can the agent work without external network access?
Yes. The environment is deployed inside the perimeter: the browser, model, and task queue do not access the external network, and exchange with internal systems goes through an approved gateway with a request log.
What happens if a scenario step fails?
The scenario is divided into idempotent steps with save points: on failure, the agent returns to the last successful one and repeats it in the same context. Payments, deletions, and document sending stop for operator confirmation.
Who is responsible for the correctness of actions?
A log is written for every action: screenshot, markup snapshot, field values, time, and result. It is used to reconstruct the cause of any discrepancy, and validation rules are set by the customer — the agent executes the scenario and does not make business decisions.
Such an environment passes internal audit without exceptions: every action has an author, time, and confirmation.What is needed to start and how we estimate the scope of work
To come back with an estimate rather than a queue of clarifying questions, we need a minimum set of input data. Below is what we usually request at project start.
- Target scenarios: which operations we are handing to the agent, in what order they currently go, and how much time they take for people — this is the basis for estimating the scope of work.
- Access and test benches: a test environment with the same configuration and permissions as production, separate accounts for the agent, and a copy of data without personal information.
- Source systems: interfaces, exports, file formats, and the rules by which data enters and leaves the process.
- Restrictions: what the agent must not do — mandatory approvals, regulations, areas where a signature or human decision is required.
- Responsible people on your side: who answers domain questions and accepts the result at each stage.
- Acceptance criteria: how we measure the result — share of automated operations, processing time per request, error rate on a control sample.
With this set, we analyze the flow of operations, mark the sections that require computer vision and those where calling interfaces is sufficient, and come back with an estimate and implementation plan: stages, timelines, and control points.
If some data is not yet available, that does not block the start — we discuss the task, record assumptions, and resolve them as work progresses.
