Embedded AI: where the model breaks on real hardware
Laboratory accuracy and on-device model behavior are two different things. In embedded AI, the model runs where there is no margin for power, memory, or time, and input comes from real sensors rather than a prepared dataset. Below are failure points that our engineers most often see when deploying a model to a device.
The first and most underestimated is an unstable communication channel. A device cannot constantly contact the server: the connection drops, the response arrives with a delay, and some events are lost.
If the logic is built on constant exchange with the cloud, the system stops responding exactly when a response is needed. That is why the loop is split: critical processing is done locally, while heavy processing goes to the server when the channel is available.
The second is the latency bottleneck. The time budget for one inference determines everything else: which branch of the model to keep, where to apply quantization, what to move to a separate thread.
Trying to preserve the original quality without accounting for this budget creates a queue at the input and increased power consumption, meaning the device heats up and discharges faster than intended in the design.
The third is model quality divergence. On the bench, metrics hold; in the field, they drop: a different sensor, noise, different lighting, shifted data distribution.
We detect this in advance — we collect a control set from real recordings and compare the result before rolling out to the entire fleet of devices.
For deployment we use ONNX Runtime and native platform tools so that the same model runs both on the server and on the device without rebuilding. Next, we cover how this is addressed: control sets, on-device profiling, and a degradation plan for a weak channel.
What on-device model inference provides
Local inference on the device moves processing inside the loop: sensor data does not leave the board, and the decision is made where the event occurred.
Autonomous operation without a network becomes a normal mode rather than an emergency one — the device continues to recognize, detect, and filter the signal without contacting an external service.
Predictable response time is a consequence of the same decision: there is no waiting for a response over the channel, no queue on the server side, and no latency growth under peak load on the fleet.
Processing one frame takes the same time as when idle, and run-to-run variation stays within a few percent.
The class of failures also changes. Channel loss, bandwidth degradation, and unavailability of an external service cease to be a function failure — they remain an entry in the event log.
What must be handled locally is different: overheating, power dips, sensor degradation, corruption of model weights in memory. These are covered by watchdog checks, integrity control, and rollback to the previous version.
Channel load stops depending on the volume of raw data: aggregates, events, and state telemetry go out instead of a raw stream. As the fleet scales, bandwidth and traffic grow more slowly than the number of devices.
Before launch, four things should be checked: model quality on the target hardware, not on a demo bench; memory and compute headroom for peak load; behavior during degradation and the method for updating the model across the device fleet; and acceptance metrics.
Embedded AI: comparison of deployment modes
The three modes are not maturity levels but different points on the scale of "how much computation the site allows" and "where data may be stored." The choice is dictated not by team preferences but by the power and memory budget of the node, the stability of the channel to the center, and requirements for what may leave the site at all. Below is a comparison by applicability and limitations.
| Mode | Where data is computed and stored | When it fits | Limitations |
|---|---|---|---|
| On-device deployment | The model and data do not leave the controller; the buffer is stored locally | Strict requirements for response latency, no stable channel, data cannot be exported | Node resource constraints: the model must be compressed and simplified, updates are rolled out on site |
| Site gateway | The gateway computes and stores buffers and archives; aggregates and events go to the center | Multiple devices on site, a shared archive is needed, and one model must be reused | The gateway is a single point of failure; power redundancy and a synchronization procedure with the center are needed |
| Centralized processing | All data goes to the center, where models are stored and retrained | The fleet is small, the channel is reliable, models change frequently | Dependence on the network: if it breaks, the site goes blind; traffic and privacy requirements increase |
The modes are not mutually exclusive: critical checks remain on the device or gateway, while heavy analytics and retraining take place in the center.
Moving between them does not require rewriting the model — the execution loop changes: the same computational graph is transferred to another node, and data routing is configured as a separate layer.
How the project proceeds: from model audit to handover
The implementation order for embedded solutions differs from server-side: the model is constrained by the device's memory, thermal envelope, and power, and updating it on already shipped hardware is more expensive than building anew.
Therefore, we complete the assessment and deployment scheme before the first line of integration code is written. Below is what the project looks like as a whole.
- Model and data audit. We examine what has already been trained, what the model's inputs and outputs are, and where it was computed before us. We check whether it can be transferred to the target hardware without loss of quality and record the metric below which the solution no longer makes sense.
- Deployment scheme. We determine where inference is computed — on the device itself, on the gateway, or in hybrid mode. We define behavior when communication is lost, the exchange format, the procedure for updating weights, and rollback to the previous version.
- Tuning and integration. We compress the model to the constraints of the specific platform and integrate it with the existing firmware or application. This also shows whether we meet the device's latency, heating, and power consumption limits.
- Pilot batch validation. We assemble a limited batch, run it on real field data, compare it with the original accuracy, and calculate the false positive rate. Based on the results, we choose: scale, fine-tune, or change the formulation.
- Production launch. Updates go to serial devices in waves, with canary rollout to part of the fleet. Telemetry on quality and failures comes from the field, so regressions are caught before they affect the entire fleet.
- Solution handover. We deliver not just a ready build: model update procedure, reproducible build, interface documentation, and rollback instructions. Your team then retrains the model and releases updates without us.
Delivery package: artifacts, diagrams, procedures
The delivery package is not an archive of files but a working system that your team continues to maintain itself. We validate every artifact in practice: the customer's engineer makes a change to the model or rule and runs it through the same loop as we do. Below is what remains in your hands after acceptance.
- Build and deployment configurations: reproducible firmware and model builds, environment variables, scripts for rollout to target devices.
- Data flow diagram: the path from sensor through preprocessing to model inference and back, with notes on where data is buffered and what happens when communication is lost.
- Access policies: roles, keys, segment boundaries — what the device exposes externally, what it computes locally, and who is authorized to replace the model.
- Model update procedure: the procedure for releasing a new version, validation on the reference set, rollback criteria, and storage of previous builds for quick return.
- Architecture documentation: module boundaries, dependencies, resource limits, and known limitations of the solution.
- Team training: code walkthrough, collaborative editing, and transfer of debugging procedures directly on the device.
- Metrics and checks log: what we measure and how often, and which deviations are considered grounds to roll back an update.
Handover is complete when your engineers can release a new model version and return to the previous one without our involvement.
Case study: quality control line moved to local inference
The customer is a production facility with three bottling lines. Quality control relied on selective operator inspection and a camera that sent frames to a server outside the shop floor.
With a narrow channel, frames accumulated in the buffer, and the verdict on a bottle arrived when it was already on the pallet: recalling it from shipment meant manually re-sorting the batch.
The missed defect rate held at 3-4%, and an inference failure was noticed by a drop in throughput — on average after 6-8 minutes.
What we did
We moved the model to an industrial NVIDIA Jetson controller right at the conveyor and closed the frame stream inside the shop floor: not a single frame leaves the site, and only the event log and aggregated statistics go out.
The confidence threshold was tuned to the real defect classes so as not to reject good product, and the signal to the rejector is sent within fractions of a second — defects are removed from the belt before packaging.
What changed
- Missed defects in shipment — from 3.8% to 0.7%: a verdict is made for every unit, not for a shift sample.
- Reaction time to failure — from 6-8 minutes to 2-3 seconds: an accelerator failure is visible immediately, and the line stops by signal rather than after an empty belt.
- Data volume leaving the site — from the full video stream to the event log: only defect labels and counters go to external storage.
- False rejections — down by a third after calibrating the threshold on accumulated line defects.
The metrics were collected on one section with its own defect set — another line will have its own numbers, so we measure before and after on real shifts.
How on-device inference differs from gateway processing
The difference is where the model executes. With on-device inference, the network is computed directly on the board next to the sensor — only the result goes out: a label, an event flag, coordinates.
With gateway processing, the device sends the raw stream over the local network, and a more powerful node handles the computation.
Latency and autonomy are decided by the device: the response fits within a single frame, and the logic continues to work when communication is lost — this is important for interlocks and emergency stops.
The price is a trimmed model: quantization, strict memory limits, selection of architectures for a specific accelerator, which is what environments like TensorFlow Lite do.
The gateway allows a larger model and can merge data from multiple sensors, but it adds dependence on the communication channel and node power.
Data privacy. With on-device inference, raw data does not leave the enclosure — only events go over the network, so video and biometrics do not require a separate processing loop.
On the gateway, the raw stream leaves the device, and if the gateway has a cloud connection, the personal data boundary expands to the entire site.
Model updates are the second fork. On the device, the model changes together with firmware or a separate image, so two slots, integrity checks, and rollback are needed: otherwise an unsuccessful update leaves the fleet inoperable, and visiting nodes takes time.
On the gateway, the model is updated centrally, but there is a requirement for input data compatibility with firmware already deployed in the field.
In practice, a hybrid is chosen more often: a lightweight detector on the device filters out the background, and the gateway clarifies disputed cases.
The final solution is dictated by latency, autonomy, privacy requirements, and the size of the fleet that will have to be maintained with every model change.
Frequently asked questions: limitations, risks, support
Before starting, customers ask the same questions: will accuracy remain after transferring the model to the device, will the solution fit the equipment already installed, and how to update a fleet without a network.
The limitations of embedded AI — memory, thermal conditions, power — appear immediately, so we answer each point before work begins.
Accuracy drops after transfer — how do you maintain it?
Quantization speeds up inference but changes the distribution of results. We run the reference dataset through the original and compiled model and compare the divergence by each class, not the average across the sample.
If degradation exceeds the agreed threshold, we change the quantization scheme or keep sensitive layers at full precision.
Will the solution fit the equipment we already have?
First, an inventory: which accelerators and resources are actually available across the fleet, not just stated in documentation. Some tasks are covered without replacing hardware — the model is converted to a more compact format.
If there is no accelerator, we compute inference on the central processor and leave preprocessing on the device. We provide a compatibility conclusion before work begins.
How do you update models on devices without a network?
For a fleet without a permanent channel, updates follow a two-copy scheme: the new version is written next to the working one and activated after an integrity check. An unsuccessful update is rolled back without visiting the site.
Weights and trigger thresholds are supplied as separate packages, so editing rules does not require rebuilding the model.
Support is built around input data drift: metrics on new device batches are checked, and rebuilding is triggered by changes in site conditions rather than by the calendar.Let's discuss the task and come back with a work plan
We start the discussion not with a presentation but with an analysis of the task: what the target platform is, what its limitations are for memory, power, and thermal conditions, and whether the model is computed directly on the device or near it.
This determines whether the solution will fit into quantization and weight pruning or will require a hybrid loop with part of the computation offloaded.
To make the estimate engineering-based rather than approximate, we need at minimum from you:
- target platform and its limitations: available memory, power, thermal conditions of the enclosure;
- task class: recognition, object detection, prediction from sensor readings, signal processing;
- what data already exists: raw recordings, labeled sets, sensor logs, their volume and annotation quality;
- requirements for autonomy and event response time, device operating mode;
- operating conditions and the class of failures that are unacceptable at the site;
- who on your side makes architecture decisions and signs off on stage acceptance.
Based on the conversation, we return not with general words but with a working package: a work plan with stages and acceptance points, a timeline estimate for each stage, and the team composition with areas of responsibility — model, embedded part, acceptance testing.
We separately describe the pilot project scenario: which node is checked first, on which sample, and by which indicator we consider the result achieved. Plus a list of risks with an honest statement of where model accuracy runs into platform resources.
Describe the task in free form — what the device is, what it should understand, and under what conditions it should operate. We will come back with a timeline estimate and work plan.
