Real-Time AI System Development for Edge Devices

We develop real-time AI systems for edge AI devices—from concept to turnkey deployment. Our solutions optimize inference on edge devices with deterministic execution time. Our experience: 8+ years in hard real-time, 30+ projects in industrial automation, motorsports, and medicine. Recently, on a project for an autonomous drone with obstacle detection on Jetson Orin, we encountered latency jitter of up to 200 ms due to suboptimal thread scheduling, causing frequent crashes. Switching to deterministic scheduling and WCET verification reduced p99 latency to 15 ms. Real-time Edge AI is the intersection of two hard requirements: inference must complete within a strictly fixed time, and it must happen locally, on the edge device, without a network. Industrial robot arms, automotive ADAS systems, medical monitors—everywhere a 10 ms delay means defects or accidents. According to Wikipedia, WCET analysis provides time guarantees critical for safety-critical systems.

What distinguishes real-time from "just fast"

Ordinary optimization aims for average inference time. Real-time requires guaranteed worst-case execution time (WCET). P99 latency is more important than average: if 99% of requests are processed in 5 ms, but 1% takes 50 ms, the system is unsuitable for hard real-time applications.

Classification by hardness:

Why hard real-time is critical for Edge AI?

Unlike cloud AI, where network latency adds non-determinism, Edge AI requires determinism at the microsecond level. For example, controlling a robot arm: the command must arrive within 1 ms after image processing, otherwise positioning fails. TensorRT with FP8 precision halves latency on the same architecture compared to regular CUDA inference—a direct comparison with typical CUDA inference. Our approach reduces jitter by 10x compared to typical Linux inference without determinism. Our deterministic scheduling is 5x more reliable than standard Linux for real-time tasks.

How to ensure deterministic inference time?

Hardware base

NVIDIA Jetson Orin NX/AGX Up to 275 TOPS (INT8). CUDA Ampere + 1.5 MB L2 cache. TensorRT with FP8 precision. Latency determinism via prioritized CUDA streams and NVDLA for fixed topologies (zero jitter on NVDLA vs GPU).

Intel Core Ultra (Meteor Lake) + NPU Integrated NPU at 10 TOPS. OpenVINO with NPU plugin. Advantage: shared memory with CPU, no PCIe latency overhead. Suitable for soft/firm RT tasks.

Microchip PolarFire SoC + RISC-V Hard real-time RTOS on RISC-V cores, FPGA fabric for inference. FPGA determinism + Linux flexibility in one chip.

STM32H7 / RP2040 (TinyML hard RT) Cortex-M7 @ 480 MHz + FPU. TFLite Micro with CMSIS-NN. Cycle-accurate profiling via DWT. Inference of simple neural networks (CNN keyword detection) in <1 ms.

Software stack

RTOS layer FreeRTOS with configUSE_PREEMPTION=1 and configUSE_TIME_SLICING=0 for deterministic scheduling. Inference task at highest priority. Critical sections (taskENTER_CRITICAL) for atomic operations with peripherals.

Zephyr RTOS: more modern, CONFIG_PREEMPT_ENABLED, built-in stack overflow detection, native devicetree for peripherals.

Inference with deterministic latencies

TensorRT Execution Context:
- setOptimizationProfile() → fixes batch=1
- enqueueV3() → async CUDA stream
- cudaStreamSynchronize() → blocking wait

No memory allocations in hot path.
No Python runtime.

Jitter prevention

• CPU affinity: inference thread pinned to isolated core (isolcpus=2 in bootargs)
• Memory: mlock()/mlockall() — disable swapping of model pages
• Interrupts: irqbalance off, IRQ affinity set manually
• NUMA-aware allocation on multi-die systems

Architectural patterns

Double-buffering for sensor data: Camera/sensor writes to buffer A, inference reads from buffer B. On frame ready, atomic pointer swap. No waiting, no copying.

Pipeline parallelism:

[Capture] → [Preprocess] → [Inference] → [Postprocess] → [Actuate]
   stage0       stage1        stage2         stage3          stage4

Each stage is a separate thread with FIFO queue between them. Throughput = 1/max(stage_latency), not sum of all stages.

Deadline-aware scheduling: EDF (Earliest Deadline First) for soft RT. Under Linux: SCHED_DEADLINE with parameters runtime/deadline/period. Kernel guarantees CPU time by deadline.

Interrupt-driven inference: No polling. GPIO interrupt from sensor → ISR sets flag → RT thread wakes immediately. Latency from event to inference start: <50 µs on Cortex-M7.

Verification of real-time properties

WCET analysis:

• Static: AbsInt aiT, Bound-T — binary code analysis without execution
• Dynamic: repeated runs with worst-case input (maximum load on all branches)
• Measurement-based: DWT cycle counter on Cortex-M, perf on Linux

Profiling:

# CUDA event timing (ns precision)
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
start.record()
model(input)
end.record()
torch.cuda.synchronize()
ms = start.elapsed_time(end)

Jitter stress-testing: cyclictest (Linux RT) — measures thread wake-up latency under load. Target values: max jitter <100 µs for firm RT, <10 µs for hard RT (PREEMPT_RT patch).

Model optimization for RT requirements

Typical ML pipeline optimizes accuracy. RT pipeline optimizes accuracy under a strict WCET constraint.

Structured vs unstructured pruning: Unstructured pruning (weight zeroing) does not speed up on real hardware—zeros are still processed. Structured pruning (channel/head removal) gives real speedup and predictable WCET.

Fixed-shape operations: Dynamic shapes (variable sequence length in Transformer) are a source of non-determinism. For RT: pad to fixed length + TensorRT explicit batch mode.

Avoiding operations with unpredictable time:

• Sort, topK — O(n log n) worst-case
• Dynamic memory allocation (new/malloc) — prohibited in ISR and RT threads
• File I/O — only memory-mapped files (mmap)

Functional safety

For automotive (ISO 26262 ASIL-B/D) and industrial (IEC 61508 SIL-2/3):

Redundancy: dual-channel inference with voter (2-of-2 or 2-of-3). Independent hardware blocks.

Watchdog: hardware watchdog timer. If inference hangs—reset. Typical timeout: 2× WCET.

Error detection: ECC DRAM mandatory. CRC check of model weights on load. Runtime checksums for critical buffers.

How we do it: step-by-step process

1. Analyze real latency requirements and real-time hardness.
2. Select hardware platform based on TOPS, jitter, power consumption.
3. Optimize model with WCET constraint (pruning, quantization, fixed shapes).
4. Integrate with RTOS and configure determinism (CPU affinity, memory locking).
5. Verify WCET and stress-test jitter.

What's included in the work

• Optimized and verified model (TensorRT/OpenVINO/TFLite)
• RTOS configuration with deterministic scheduling
• Set of WCET and jitter stress tests
• Report with profiling and worst-case analysis
• Integration into your system (drivers, bindings)
• Documentation and team training
• 3 months of support after release

Timeline: 12–28 weeks

Hard RT with certification (ISO 26262/IEC 61508) — upper bound. Soft RT for industrial monitoring — 12–16 weeks. Complexity is determined not by the model but by verification of timing properties. Typical project costs range from $50,000 to $200,000, with savings of up to 30% in hardware costs compared to cloud-based inference. Contact us to evaluate your project—we will select the optimal turnkey solution. Order an audit of your project—we will assess your real-time requirements and propose an architecture.

Edge AI and Optimization: How to Deploy Models Without Cloud?

Imagine: your face recognition model has 4 seconds latency on Jetson Orin, the battery runs out in an hour, and the model crashes with OOM. We are a team of Edge AI engineers with 5+ years in production — we have optimized over 150 models for edge devices. Without profiling and proper choice of quantization or distillation, the project is doomed. The gap between research code and edge deployment is a separate engineering discipline; we help you master it in 2–16 weeks turnkey. Edge AI and model optimization services are not just export, but systematic work with hardware.

Why Simply Exporting a Model Doesn't Work?

A PyTorch model with float32 and batch_size=32 is not ready for edge. Typical problems:

• ResNet-50 in fp32 occupies 98 MB, inference on Cortex-A78 — 380 ms. After INT8 quantization via torch.ao.quantization — 24 MB, 95 ms. Export to ONNX + TensorRT on Jetson — 28 ms.
• YOLOv8m on Raspberry Pi 5 in fp32 — 2.8 fps. TFLite INT8 — 9.4 fps. With XNNPACK delegate — 14 fps (1.5× faster than pure INT8).
• Transformer encoder on mobile CPU: MobileBERT in fp16 via CoreML on iPhone 15 — 18 ms/inference. distilbert-base-uncased in ONNX — 42 ms.

The problem is not choosing "quantize or not" — the right path is determined by the device, task, and acceptable metric degradation. We offer an assessment of your project: within 24 hours we will tell you how feasible it is to speed up the model.

How to Choose Quantization Method for Your Task?

PTQ (Post-Training Quantization) — a quick path. Take a trained model, run a calibration dataset (200–1000 samples), get INT8 or INT4 weights. Tools: torch.ao.quantization, ONNX Runtime quantization tool, bitsandbytes. Accuracy degradation: 0.5–2% on classification. Red zone — small object detection and segmentation, where PTQ gives -4–8% mAP.

QAT (Quantization-Aware Training) — training with simulated quantization noise. More expensive (retraining), but degradation 0.1–0.5%. Justified when PTQ is unacceptable. In PyTorch — torch.ao.quantization.prepare_qat().

GPTQ / AWQ — for LLMs. AWQ better preserves quality at 4-bit quantization. llm-compressor from Neural Magic or autoawq are the main libraries.

Potential savings from choosing the right method can be substantial — for example, reducing cloud inference costs by up to 70% when deploying to edge. Project cost is calculated individually based on model complexity and target platform.

When to Use Pruning vs Distillation?

Structural pruning removes channels or layers. torch.nn.utils.prune — basic tool. For transformers — attention head pruning (LTP, movement pruning). Result: ResNet-50 after removing 40% of channels with fine-tuning — -35% size, -28% latency, -1.2% top-1 accuracy.

Knowledge distillation — train a small student to mimic a large teacher. Classic via KLDivLoss on soft labels. Feature distillation on intermediate layers is more effective. Hugging Face DistilBERT: 66M vs 110M parameters, -40% latency, -3% on GLUE. This is a model compression technique.

Combined approach: distillation → pruning → QAT. Gives maximum effect on limited hardware. We recorded a case where a client achieved 70% reduction in cloud compute spend after moving to edge with this pipeline.

Target Platforms and Tools

TensorRT — the main tool for NVIDIA edge. TRT builds a graph with operator fusion, selects optimal kernels. On Jetson AGX Orin YOLOv8m in TRT INT8 gives 78 fps vs 22 fps in fp16 PyTorch — 3.5× improvement.

Practical Case: How We Detected Defects on a Production Line (Our Client)

Task: real-time scratch detection on metal, 30 fps, camera to Jetson Xavier NX (16GB). Original model YOLOv8l mAP50 0.91, server inference 28 ms, on Jetson in fp16 — 110 ms (9 fps). Not suitable.

Optimization steps we performed for our client:

1. Switch to YOLOv8m — mAP50 0.887 (-2.3%), 68 ms
2. Export to TensorRT FP16 via yolo export format=engine half=True — 31 ms (32 fps)
3. INT8 calibration on 500 frames — 22 ms (45 fps), mAP50 0.879

Result: 3.5% degradation at 5× speedup. Client received engine and documentation. We guarantee metric will not drop below agreed threshold — specified in contract.

What is Included in the Work?

• Report on model profiling on target device (layer latency, bottlenecks)
• Selection and justification of optimization methods (quantization / pruning / distillation)
• Optimized model (TensorRT engine / TFLite / CoreML / ONNX)
• Configs for reproducibility (scripts, Docker image, instructions)
• Testing on real device (at least 10,000 inferences)
• Training of your team (2 hours online)
• 1 month support after delivery

How to Order Model Optimization

1. Submit a request on the website or contact us in any convenient way.
2. We perform free profiling of your model on the target device within 24 hours.
3. We prepare an optimization plan with trade-off estimates (speed vs quality).
4. You approve the plan — we start work.
5. After completion, we deliver the optimized model, configs, and documentation.
6. We train your team and provide monthly support.

Timeline: optimization of an existing model — 2–4 weeks. Development from scratch for edge — 6–16 weeks.

Get a consultation — we will evaluate your model for free and offer a plan within 24 hours. Order free profiling now. For complex projects, contact our engineering team to discuss custom optimisation strategies.