Where ROS AI integration falls apart on board the robot
The perception module confidently finds the object in a test recording, but onboard the robot everything falls apart: the command to the control loop is sent using coordinates that have already become outdated.
The issue is not the quality of the model, but the interface — in ROS 2, perception and control live in different nodes and communicate through message queues, and it is at this boundary that time is lost.
The first typical gap is control loop desynchronization in time. Timestamps are set by different sources: camera driver, detection node, trajectory planner.
Clock drift produces a shift of a frame or two, and the manipulator moves toward an object that is no longer there: object loss looks like a model error, although the cause is synchronization.
The second is a frozen node. The process is alive, but frame publication has stopped, and without liveness monitoring the loop continues to compute using the last frame until the robot hits a limit stop or aborts the operation. Recovery is manual: stop the cycle, restart, recalibrate, and the section is down.
Response latency and object loss in these areas combine into a single failure class that is not visible during desk debugging.
Below we break down how this is closed: where to place node liveness monitoring, how to bring time to a single source, and what to do when an object is lost — without rewriting the control loop. Then ROS AI integration stops depending on who noticed the failure first.
What a unified ROS graph provides: perception and control in one loop
A unified ROS 2 graph is when the detector, tracker, planner, and actuator drivers exchange messages in one loop instead of living as isolated services with their own clocks and buffers.
Each topic has a quality-of-service profile: reliable delivery for commands to the actuator, and no retransmission of outdated frames for the video stream. Observability is end-to-end — from the log you can see at which node a frame became a target and where the target was lost.
The first effect is an eliminated failure class: "the robot did not see the object." It was born at the interface: timestamp desynchronization, a lost frame, a queue that the control module was clearing with already outdated data.
In a shared graph, timestamps come from a single source, and buffer overflow is visible via a diagnostic message before the robot stops.
The second is the share of manual interventions: instead of restarting individual processes, the operator watches the faulty node undergo a lifecycle transition and return to operating mode.
Downtime duration and recovery time after a failure are now determined by restart parameters rather than by the reaction speed of the operator on duty.
This assembly relies on a mature ecosystem of drivers and recording tools: an incident is reproduced from logs rather than guessed from hardware behavior. That is why we conduct model integration as an engineering process with a verifiable result.
Perception module operating modes: onboard, station, external loop
Placement of computation in the perception loop is a choice of failure boundary: what the robot will continue to do if the channel to the station is lost. It determines the model size that fits within the required response time and the amount of data leaving the robot.
There are three operating modes: onboard loop, station-hosted loop, and hybrid scheme. In each, ROS 2 transport is configured differently and roles are distributed among nodes.
| Mode | When it fits | Mode limitations |
|---|---|---|
| Onboard loop | Autonomous operation, movement outside reliable communication range, predictable response | Ceiling on the number and size of models, thermal budget, model update — with node restart |
| Station-hosted loop | Heavy models, fine-tuning on a live stream, multiple robots on a shared pool | Dependency on the channel: uneven latency, losses, reconnection |
| Hybrid scheme | Fast response plus semantics: onboard — safety, on the station — recognition and planning | Model version alignment, time synchronization, two sets of configurations |
We separately decide what is published externally: raw video stream and point clouds do not go over the channel — decimation and compression remain onboard.
We transmit control commands with delivery confirmation and a short queue; the frame stream — without retransmissions, otherwise the robot reacts to outdated data.
We fix the boundaries of the modes in documentation: which loop publishes what and at what channel degradation the scheme switches. Then the model can be transferred between onboard and station without rewriting the logic.
Implementation stages: from bench survey to handover to operation
We start the work with a survey of the bench: we capture a map of ROS 2 nodes, topic publication rates, available compute units, and where time is currently lost — at frame capture, inference, or command transmission to the actuator.
This map shows what can be reused and what will have to be rewritten; without it, any estimate turns into a guess.
- Bench survey. Inventory of nodes, topics, and services, measurement of actual rates and latencies, check of power and compute units, list of site limitations.
- Graph design. We design the graph: who publishes and who subscribes, where buffers and message delivery policies are placed, how perception and control loops are separated so that the failure of one node does not stop the platform.
- Node configuration. We assemble and configure components, move parameters into files, enable telemetry recording and replay mode — we debug on it without traveling to the site.
- Testing at the proving ground. We run scenarios with repeatable conditions, measure latency from frame to command and the hit rate, and catch degradation under glare, vibration, and communication loss.
- Pilot operation. The system runs at the site under engineer supervision: we analyze every failure via logs, adjust thresholds and switching logic.
- Handover to operation. Graph design, configurations, startup and recovery procedures, shift training, and an agreed rollback plan to the previous configuration.
Each stage is closed with an artifact that can be accepted separately: survey protocol, graph design, configurations, proving ground run report, pilot operation log.
This way, the transition to the next step does not look like a leap into the unknown — the customer always has the current state of the system in hand.
Handover package: configurations, graph designs, procedures, and access rights
A ROS 2 project is handed over not with a demonstration of a working robot, but with a set of artifacts by which your team can operate the system without us.
If handover comes down to an archive with source code, in a couple of months no one will remember why the motor control node publishes commands with such a delay and what will happen if the trigger threshold is changed. Below is what exactly remains with the customer.
- Node configurations — parameter files for each node: publication rates, queue sizes, trigger thresholds, binding to topics and namespaces. Alongside — comments on which values were changed under load and why.
- Graph designs — a diagram of nodes, topics, and services with QoS profiles and points where the camera data stream diverges from the control command stream.
- Operating procedures — the order of standard startup and shutdown, response to loss of communication with the camera or controller, signs of an incident, and operator actions before the engineer arrives.
- Instructions for the support service — how to read logs, what diagnostic messages mean, which metrics to capture when response slows down.
- Access rights — role model: who can change configuration, who can only observe, who has rights to restart nodes and access logs.
- Rollback procedure — versioning of configurations and a way to return the previous working parameter set with one command.
- Traceability matrix — mapping of requirements to nodes and checks, so that during refinement it is visible what exactly is affected.
We consider the package accepted when your engineer independently deploys the system, starts it, and takes it out of operation according to the documents, without asking us for explanations.
Case study: how the line stopped halting due to perception failure
On a machine vision line, a defect detector ran in a ROS 2 loop and made perception errors every shift: glare from metal and changes in lighting were read as defects.
The line halted on a false alarm, and the operator went to the control panel and manually confirmed frames that the system could not interpret.
Out of ten stops per shift, seven were due to perception rather than a real defect. Cycle recovery took up to twelve minutes of downtime, and manual interventions amounted to more than twenty per shift.
What was changed
We rebuilt the perception loop: cameras were synchronized by timestamp, a frame buffer and recalibration for current lighting were added, and a confidence threshold was introduced — below it, a frame goes for review rather than to a stop. Emergency stop was replaced with a safe reduced-speed mode until operator confirmation.
What the result was
| Metric | Before | After |
|---|---|---|
| Line stops due to perception, per shift | 6–8 | 0–1 |
| Cycle recovery time | up to 12 min | up to 2 min |
| Manual operator interventions per shift | 20+ | 2–3 |
The operator stopped being a mandatory link in the cycle — intervention remained for real defects. Disputed frames now accumulate in a dataset for model fine-tuning, so the confidence threshold and calibration are refined based on actual data, without stopping the line.
What to do if your own perception module is already deployed?
A working perception module is an asset, not an obstacle: we do not rewrite it from scratch but build the integration around it.
First of all, we fix the contract: in what form the module returns the result (message, set of coordinates, object label), in which coordinate system, at what rate, and with what latency.
This is exactly where the cause of desynchronization is most often revealed — the nodes are formally compatible, but live in different reference frames and different time.
Next, we check node compatibility not on the line but on recorded logs: we run a day of real data through the new scheme and look at discrepancies for each object class. This gives an answer before intervening in the working loop.
We build the gradual transition by subsystems: first the module runs in shadow mode and only records the result, then control is switched to it for one section of the route, then for the rest.
Each step is reversible: return to the previous scheme is one command, without reconfiguring equipment and without stopping the line.
The output is migration to the new scheme with a fixed set of checks: the share of paths covered by verification, switching time, and rollback procedure are described in the documentation.
Describe what you already have deployed, and we will come back with an estimate and a phased migration plan.
Doubts before starting: we break down technical risks
Most questions before starting are not about the model, but about how a ROS 2 node will coexist with an already working loop. These are the main technical risks: compatibility, behavior during failures, and latency in the loop.
Will sensors and controllers have to be changed?
No. The node subscribes to already published topics and sends commands in the usual message format, so drivers and sensors remain in place. If a sensor does not have a ready-made driver, we write an adapter — the integration check runs on your hardware, not on the bench.
What happens if communication is lost or a node fails?
A watchdog timer remains in the loop: when the data stream is lost, the node releases control and puts the system into a safe mode instead of continuing motion based on outdated data.
We separately verify that after communication is restored, the loop returns to operation without manual restart.
How do you check latencies in the loop?
We record telemetry along the entire chain — from the camera frame to the control command — and look at the latency distribution, not the average.
We run scenarios under real conditions: low lighting, channel congestion, unexpected obstacles — this way bottlenecks are visible before handover.
What happens after handover?
We provide failure diagnostics documentation: which metrics to watch, where event logs are stored, what each error means. Post-handover support includes incident analysis and refinement of procedures — you are not left alone with the loop after launch.
Let's discuss the task and come back with an estimate and work plan
An estimate based on the description "we need recognition on cameras" always turns out approximate.
To calculate the scope of work, we need the current node graph, the set of sensors, and requirements for the control loop — from them it is clear whether perception fits into existing topics or will require a separate communication branch.
What to send for an estimate:
- platform description: compute module, accelerator, installed ROS 2 branch;
- set of sensors with interfaces and publication rates: cameras, lidar, inertial unit, odometry;
- current node graph — which topics and services already exist, who publishes and who subscribes;
- requirements for loop response: permissible latency, control rate, behavior on frame loss;
- scenarios that the solution must handle, including difficult ones — poor lighting, occlusions, dynamic obstacles;
- recordings of bench runs and logs, if they are kept.
The work plan will include: a design for integrating perception with the existing graph, selection of the model and data exchange format, nodes with message delivery policies, bench tests, and acceptance on the customer's scenarios. We fix the order of stages in advance so that each step has a verifiable result.
Describe the task and send the materials — we will come back with an estimate, stage order, and launch conditions.
