Deploying ML Models on Edge with AWS IoT Greengrass

Deploying ML Models on Edge with AWS IoT Greengrass

Typical scenario: a factory with 200 PLCs and cameras, but the model was trained in SageMaker. Manually updating binaries on each device leads to errors and downtime. We configure AWS IoT Greengrass v2 — a runtime for running ML components on Edge without SSH. Models from SageMaker are deployed to the thing group "factory-line-1", with centralized fleet management via zero-touch provisioning. Over 5 years, we have deployed Greengrass on 30+ industrial sites and guarantee stability under network interruptions.

Greengrass v2 Architecture

Nucleus — the Greengrass core runtime in Java. Manages component lifecycle, IPC, and the local MQTT broker (Moquette). Components are atomic deployment units: AWS managed (ready-made) and custom (your code). A deployment is a JSON document with a set of components and versions. The AWS console or CLI sends it to the device, and Nucleus applies it.

Cloud (AWS): SageMaker → Model artifact in S3 IoT Greengrass Deployment → targets: thing group "factory-line-1"

Edge device: Greengrass Nucleus ├── aws.greengrass.SageMakerEdgeManager ├── aws.greengrass.DlrInferenceService (DLR runtime) └── com.company.InferenceApp (custom component)

Why Greengrass for Edge ML?

Centralized management — just update the deployment and all 500 devices get the new model. Staged rollout with canary (first 5%, then 100%) and automatic rollback on error. Device Shadow stores the current model version and status, syncing cloud ↔ edge even during temporary network disconnects.

ML Inference Components

SageMaker Edge Manager compiles models via SageMaker Neo for specific hardware (ARM, x86, GPU). Optimized binaries plus an Edge Agent (gRPC daemon) run on the device.

# edge agent gRPC client
import agent_pb2, agent_pb2_grpc
channel = grpc.insecure_channel('unix:///tmp/sagemaker_edge_agent_example.sock')
stub = agent_pb2_grpc.AgentStub(channel)

# load model
stub.LoadModel(agent_pb2.LoadModelRequest(
    url=f's3://bucket/model.tar.gz',
    name='defect-detector'))

# inference
response = stub.Predict(agent_pb2.PredictRequest(
    name='defect-detector',
    tensors=[tensor]))

DLR (Deep Learning Runtime) — an alternative to Edge Manager. TVM-compiled models, supports TFLite, XGBoost, LightGBM. Lighter, without gRPC overhead. On CPU, DLR is often 2x faster.

import dlr
model = dlr.DLRModel('/greengrass/v2/work/model', 'cpu')
result = model.run({'input': numpy_array})

Comparison of SageMaker Edge Manager and DLR

How to Organize Model Updates Without Downtime?

Classic pain point of Edge ML — how to update the model on 500 devices. Greengrass solves this natively.

According to AWS documentation, Greengrass v2 supports staged rollout with automatic rollback.

Workflow:

1. SageMaker trains a new model version.
2. Artifact is uploaded to S3.
3. Component version in Greengrass is updated.
4. A new Deployment is created on the thing group.
5. Nucleus on each device downloads the delta, applies it, and rolls back on error.

Staged rollout: Deployment with canary — first 5% of devices (thing group subset), CloudWatch metric monitoring, then 100%.

Local shadow for state: AWS IoT Device Shadow stores the current model version, inference status, and custom metrics. Cloud ↔ edge sync works even during temporary network disconnects.

Custom Inference Component

recipe.json:

{
  "RecipeFormatVersion": "2020-01-25",
  "ComponentName": "com.company.QualityInspection",
  "ComponentVersion": "1.0.0",
  "ComponentDependencies": {
    "aws.greengrass.DlrInferenceService": {"VersionRequirement": ">=1.6.0"}
  },
  "Manifests": [{
    "Platform": {"os": "linux"},
    "Lifecycle": {
      "Install": "pip3 install -r requirements.txt",
      "Run": "python3 {artifacts:path}/inference.py"
    },
    "Artifacts": [
      {"URI": "s3://my-bucket/inference.py"},
      {"URI": "s3://my-bucket/model.tar.gz", "Unarchive": "ZIP"}
    ]
  }]
}

Stream Manager for Large Data

greengrass.StreamManager — a local buffer for edge → cloud transmission. When the network is unstable, data is buffered locally (up to 256 MB) and automatically sent to Kinesis/S3/IoT Analytics when connectivity recovers. Parameters are configurable: buffer size, export frequency, overflow policy.

Fleet Management

Thing Groups + Attributes: factory-line-1, factory-line-2 — different models for different lines. Attribute: firmware_version, camera_model — for conditional deployment.

Greengrass CLI for diagnostics:

greengrass-cli component list          # running components
greengrass-cli logs get --component com.company.QualityInspection
greengrass-cli deployment create ...   # local deployment for debugging

CloudWatch metrics: standard ComponentRunning, ComponentBroken. Custom via EMF (Embedded Metrics Format) — any metrics from edge code directly into CloudWatch.

Implementation Plan and Timelines

What's Included

• Audit of current infrastructure and model (compatibility with Greengrass, hardware requirements).
• Architecture design: choice between SageMaker Edge Manager and DLR, component composition, deployment scheme.
• Setup of Greengrass Nucleus and components (AWS managed + custom).
• Integration with existing systems (SCADA, MES, Kafka).
• Staged rollout and monitoring (CloudWatch, alerts).
• Operations documentation and team training.
• 30-day stability guarantee post-launch.

Supported Platforms

Linux (arm64, armv7, x86_64). Minimum: 512 MB RAM, 1 GHz CPU. Tested: Raspberry Pi 4, NVIDIA Jetson, BeagleBone, industrial x86 IPCs. Windows (x86_64): Greengrass v2.9+, limited component set.

Timelines and Cost

First Greengrass deployment with a ready model — from 1 week. Custom components, fleet management, staged rollout, integration with existing SCADA/MES — 4–6 weeks. Cost is calculated individually after an audit. Evaluate your project — contact us. Using Greengrass and want to reduce operational costs? Request a consultation — we'll choose the optimal architecture.

Edge AI and Optimization: How to Deploy Models Without Cloud?

Imagine: your face recognition model has 4 seconds latency on Jetson Orin, the battery runs out in an hour, and the model crashes with OOM. We are a team of Edge AI engineers with 5+ years in production — we have optimized over 150 models for edge devices. Without profiling and proper choice of quantization or distillation, the project is doomed. The gap between research code and edge deployment is a separate engineering discipline; we help you master it in 2–16 weeks turnkey. Edge AI and model optimization services are not just export, but systematic work with hardware.

Why Simply Exporting a Model Doesn't Work?

A PyTorch model with float32 and batch_size=32 is not ready for edge. Typical problems:

• ResNet-50 in fp32 occupies 98 MB, inference on Cortex-A78 — 380 ms. After INT8 quantization via torch.ao.quantization — 24 MB, 95 ms. Export to ONNX + TensorRT on Jetson — 28 ms.
• YOLOv8m on Raspberry Pi 5 in fp32 — 2.8 fps. TFLite INT8 — 9.4 fps. With XNNPACK delegate — 14 fps (1.5× faster than pure INT8).
• Transformer encoder on mobile CPU: MobileBERT in fp16 via CoreML on iPhone 15 — 18 ms/inference. distilbert-base-uncased in ONNX — 42 ms.

The problem is not choosing "quantize or not" — the right path is determined by the device, task, and acceptable metric degradation. We offer an assessment of your project: within 24 hours we will tell you how feasible it is to speed up the model.

How to Choose Quantization Method for Your Task?

PTQ (Post-Training Quantization) — a quick path. Take a trained model, run a calibration dataset (200–1000 samples), get INT8 or INT4 weights. Tools: torch.ao.quantization, ONNX Runtime quantization tool, bitsandbytes. Accuracy degradation: 0.5–2% on classification. Red zone — small object detection and segmentation, where PTQ gives -4–8% mAP.

QAT (Quantization-Aware Training) — training with simulated quantization noise. More expensive (retraining), but degradation 0.1–0.5%. Justified when PTQ is unacceptable. In PyTorch — torch.ao.quantization.prepare_qat().

GPTQ / AWQ — for LLMs. AWQ better preserves quality at 4-bit quantization. llm-compressor from Neural Magic or autoawq are the main libraries.

Potential savings from choosing the right method can be substantial — for example, reducing cloud inference costs by up to 70% when deploying to edge. Project cost is calculated individually based on model complexity and target platform.

When to Use Pruning vs Distillation?

Structural pruning removes channels or layers. torch.nn.utils.prune — basic tool. For transformers — attention head pruning (LTP, movement pruning). Result: ResNet-50 after removing 40% of channels with fine-tuning — -35% size, -28% latency, -1.2% top-1 accuracy.

Knowledge distillation — train a small student to mimic a large teacher. Classic via KLDivLoss on soft labels. Feature distillation on intermediate layers is more effective. Hugging Face DistilBERT: 66M vs 110M parameters, -40% latency, -3% on GLUE. This is a model compression technique.

Combined approach: distillation → pruning → QAT. Gives maximum effect on limited hardware. We recorded a case where a client achieved 70% reduction in cloud compute spend after moving to edge with this pipeline.

Target Platforms and Tools

TensorRT — the main tool for NVIDIA edge. TRT builds a graph with operator fusion, selects optimal kernels. On Jetson AGX Orin YOLOv8m in TRT INT8 gives 78 fps vs 22 fps in fp16 PyTorch — 3.5× improvement.

Practical Case: How We Detected Defects on a Production Line (Our Client)

Task: real-time scratch detection on metal, 30 fps, camera to Jetson Xavier NX (16GB). Original model YOLOv8l mAP50 0.91, server inference 28 ms, on Jetson in fp16 — 110 ms (9 fps). Not suitable.

Optimization steps we performed for our client:

1. Switch to YOLOv8m — mAP50 0.887 (-2.3%), 68 ms
2. Export to TensorRT FP16 via yolo export format=engine half=True — 31 ms (32 fps)
3. INT8 calibration on 500 frames — 22 ms (45 fps), mAP50 0.879

Result: 3.5% degradation at 5× speedup. Client received engine and documentation. We guarantee metric will not drop below agreed threshold — specified in contract.

What is Included in the Work?

• Report on model profiling on target device (layer latency, bottlenecks)
• Selection and justification of optimization methods (quantization / pruning / distillation)
• Optimized model (TensorRT engine / TFLite / CoreML / ONNX)
• Configs for reproducibility (scripts, Docker image, instructions)
• Testing on real device (at least 10,000 inferences)
• Training of your team (2 hours online)
• 1 month support after delivery

How to Order Model Optimization

1. Submit a request on the website or contact us in any convenient way.
2. We perform free profiling of your model on the target device within 24 hours.
3. We prepare an optimization plan with trade-off estimates (speed vs quality).
4. You approve the plan — we start work.
5. After completion, we deliver the optimized model, configs, and documentation.
6. We train your team and provide monthly support.

Timeline: optimization of an existing model — 2–4 weeks. Development from scratch for edge — 6–16 weeks.

Get a consultation — we will evaluate your model for free and offer a plan within 24 hours. Order free profiling now. For complex projects, contact our engineering team to discuss custom optimisation strategies.