When developing a mobile app for video calls, clients often face the problem: the standard implementation of a virtual background via the server introduces delays and artifacts. In one project, the client used cloud AI from AWS Rekognition — each frame was sent to the cloud and returned after 60–80 ms. As a result, the contour fluctuated, and users complained about quality.
We solved it by moving segmentation to the device. Server-side processing adds 40–80 ms per frame, which at 30 fps causes noticeable contour breakup and a 'ghost' effect during fast movements. On-device segmentation completes in 8–28 ms, saving up to 80% of time compared to cloud inference. The key advantage is segmentation on the device, not in the cloud. This not only reduces infrastructure costs but also ensures user privacy. On each frame of the video stream, the neural network extracts the human silhouette, applies the background (image, video, or blur), and returns the result to the encoder pipeline — all without transmitting data to the server. The typical time budget is 33 ms per frame, and the on-device solution easily fits. For budget devices, we use lightweight models and reduce the frame rate to 24 fps, which ensures stable operation without overheating.
What are the advantages of on-device AI virtual background?
The task is to extract the human silhouette on each frame of a video stream (30 fps), apply the background, and return the result to the pipeline before encoding. This means a budget of ~33 ms per frame including capture, model inference, post-processing, and rendering.
Server-side: capture → send → inference → response → rendering. Even with an ideal network, roundtrip adds 40–80 ms. In practice, this means contour jitter and 'ghost' during movement.
On device: capture → inference → rendering. Everything in one pipeline. Infrastructure costs are high with the server approach — GPU servers are needed. On-device approach completely eliminates these costs. Savings on server GPU computing can reach $2,000 per month for an app with 10,000 active users. In another project, savings were $3,000 per month by abandoning expensive GPU instances. For a typical project, the investment is between $5,000 and $25,000 depending on complexity.
On-device segmentation process
We use neural networks optimized for mobile chips (Neural Engine, GPU, DSP). Inference runs locally; no data leaves the device — this simultaneously solves privacy and latency issues.
iOS: MLKit + CoreImage or Vision
On iOS we use the Vision framework with the VNGeneratePersonSegmentationRequest model. Apple added it in iOS 15 and later — it runs on the Neural Engine without explicit model loading. Accuracy is good for the front camera, but it can produce jagged contours with complex hairstyles and transparent clothing elements.
// Configure segmentation
let request = VNGeneratePersonSegmentationRequest()
request.qualityLevel = .balanced // .accurate gives better contour but is heavier
request.outputPixelFormat = kCVPixelFormatType_OneComponent8
// In AVFoundation frame handler
let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, options: [:])
try handler.perform([request])
guard let mask = request.results?.first?.pixelBuffer else { return }
// mask — CVPixelBuffer 8-bit, apply via CIBlendWithMask
CIBlendWithMask with CIContext(options: [.workingColorSpace: NSNull()]) — render in Metal, avoiding color space conversion. Without this, each frame adds ~5 ms just for conversion.
For higher quality segmentation, we convert a TFLite model like DeepLab v3 or MediaPipe SelfieSegmentation to Core ML via coremltools and load it through MLModel. MediaPipe gives a stable contour even at blurry edges. Apple Vision VNGeneratePersonSegmentationRequest
Android: MLKit Selfie Segmentation
val segmenter = Segmentation.getClient(
SelfieSegmenterOptions.Builder()
.setDetectorMode(SelfieSegmenterOptions.STREAM_MODE) // optimized for video
.enableRawSizeMask()
.build()
)
// In CameraX ImageAnalysis handler
override fun analyze(imageProxy: ImageProxy) {
val inputImage = InputImage.fromMediaImage(imageProxy.image!!, imageProxy.imageInfo.rotationDegrees)
segmenter.process(inputImage)
.addOnSuccessListener { segmentationMask ->
val mask = segmentationMask.buffer
// Apply background via RenderScript or Vulkan compute shader
applyBackground(mask, imageProxy)
}
.addOnCompleteListener { imageProxy.close() }
}
STREAM_MODE is critical — it keeps internal state between frames and runs faster than SINGLE_IMAGE_MODE. On Pixel 6 with Tensor G2, inference takes 8–12 ms. On budget devices (Snapdragon 695) — 20–28 ms. For mask post-processing, we use RenderScript (deprecated in API 31+) or Vulkan compute shader via RenderEffect on Android 12+. MLKit Selfie Segmentation
Comparison of segmentation models
| Model | Platform | Latency (ms) | Contour quality |
|---|---|---|---|
| Vision (Apple) | iOS | 12–20 | Good |
| MLKit Selfie Segmentation | Android | 8–12 | Excellent |
| MediaPipe | Cross-platform | 15–25 | Average |
| Core ML (DeepLab) | iOS | 20–30 | High |
Comparison of approaches: server vs on-device
| Parameter | Server segmentation | On-device segmentation |
|---|---|---|
| Latency | 150–300 ms (roundtrip) | 8–28 ms (inference) |
| Network dependency | Critical | None |
| Privacy | Data goes to server | Data stays on device |
| Accuracy | High (large model) | Good (optimized models) |
| Infrastructure cost | High (GPU servers) | Zero (only software) |
Background application: three options
Static image — simplest case. CIBlendWithMask on iOS, PorterDuff compositing on Android.
Blur — CIGaussianBlur filter with radius 12–20 applied to the original frame, then mask selects between original and blurred. On Android — RenderEffect.createBlurEffect (API 31+) or custom blur via Vulkan.
Video background — needs a decoder synchronized with the video call timing. On iOS — AVPlayerItemVideoOutput + Metal texture. Memory heavy: video background buffer + camera buffer + mask buffer + result. On iPhone 12 with 4 GB it's fine, on iPhone SE 2nd gen (3 GB) we need aggressive buffer reuse.
How to integrate background replacement into WebRTC pipeline?
Most mobile calling solutions are built on WebRTC — via LiveKit, Daily.co, Agora, or native WebRTC. All provide a custom VideoSource/VideoProcessor mechanism for frame manipulation before encoding.
In LiveKit SDK for iOS it's the VideoProcessor protocol:
class BackgroundReplacementProcessor: VideoProcessor {
func process(frame: RTCVideoFrame) -> RTCVideoFrame? {
// Segmentation + background application
// Return new RTCVideoFrame with processed buffer
}
}
room.localParticipant?.videoTracks.first?.processor = BackgroundReplacementProcessor()
Important: RTCVideoFrame works with CVPixelBuffer in format kCVPixelFormatType_420YpCbCr8BiPlanarFullRange. Converting to RGB for ML inference and back incurs losses. If the model accepts YUV, we keep the format untouched.
Criteria for choosing a segmentation model
Model choice depends on target devices and quality requirements. For iOS with A12+, Vision is suitable — built-in model, no extra resources needed. For Android with Tensor G2 or Snapdragon 8 Gen 1, MLKit gives the best quality. On weak devices (Snapdragon 695, A11) we use MediaPipe or lower fps.
Optimization for low-end devices
On resource-constrained devices we reduce frame rate to 24 fps and use lightweight models (MediaPipe). We also apply dynamic scaling of the input frame: reduce resolution to 480p before inference, then upscale the mask. This cuts processing time by 30-40% without noticeable quality loss.
Deliverables
After project completion, you get a fully integrated background replacement feature tested on 20+ real devices. Deliverables include:
- Source code of the segmentation and background processing module.
- Detailed technical documentation for integration and customization.
- Training of your team on the code and configurations.
- One month of support after launch to fix potential bugs.
- Recommendations for optimizing for new OS versions.
Integration timeline
- Audit of the current WebRTC stack and frame pipeline.
- Selection of segmentation model based on quality/speed (Vision, MLKit, Core ML, MediaPipe).
- Prototype development with performance measurement on 10+ devices.
- Integration into the existing WebRTC pipeline via VideoProcessor.
- Optimization of mask post-processing (antialiasing, feathering).
- Testing on edge cases (complex background, fast movements).
- Documentation and training of the client's team.
Timeline estimates
Basic implementation with background blur (one platform) — 2–3 weeks, typically costing $5,000-$10,000. Full implementation with support for static images and video backgrounds, both platforms, integration into existing WebRTC stack — 5–8 weeks, costing $15,000-$25,000.
We have 8+ years of experience in mobile development and have completed 50+ projects with video and AI features. We guarantee stable operation on devices older than five years.
Contact us for a project evaluation and get an engineer consultation.