Multi-Camera Streaming Implementation for Mobile Apps

Multi-Camera Streaming Implementation from Mobile Device

We have implemented several multi-camera streaming projects for iOS and Android. One of the key challenges is simultaneous capture from the front and rear cameras. It became technically feasible on iOS with the introduction of AVCaptureMultiCamSession in iOS 13. Before that, any "multi-camera" stream was a simulation: switching with delay or a pre-recorded second source. Now on iPhone XS and newer, you can capture both streams simultaneously — with limitations discussed below. Our 8 years of experience in mobile development and over 30 streaming projects guarantee a stable solution under load.

Custom Implementation Required for Multi-Camera Streaming

Out-of-the-box SDKs (e.g., Wowza GoCoder or HaishinKit) provide basic multi-cam but do not optimize thermal behavior or offer flexible PiP. In our implementation using Metal, we control every stage: from capture to composition. This allows adaptation to various scenarios — from webinars to remote inspection. Custom architecture delivers up to 40% more stable FPS during long broadcasts compared to boxed solutions. For authoritative reference, see Apple Developer Documentation: AVCaptureMultiCamSession.

Checking Device Support

On Android, support depends on the OEM: for example, Xiaomi Redmi Note 10 does not deliver two streams without errors. Testing on 5–10 target devices is mandatory. See also Android Developer Guide: Camera2 API.

Ensuring Stream Synchronization

Synchronization is key. Without it, the PiP window "jitters" relative to the main video. We use AVCaptureDataOutputSynchronizer on iOS and corresponding timestamps on Android. This guarantees desynchronization of no more than one frame (33 ms at 30 fps).

import AVFoundation let backOutput = AVCaptureVideoDataOutput() let frontOutput = AVCaptureVideoDataOutput() let synchronizer = AVCaptureDataOutputSynchronizer(dataOutputs: [backOutput, frontOutput]) synchronizer.setDelegate(self, queue: syncQueue) 

Architectural Limitations to Know Before Starting

AVCaptureMultiCamSession is not supported on all devices. Before initialization, mandatory check:

import AVFoundation let multiCamSession = AVCaptureMultiCamSession() guard AVCaptureMultiCamSession.isMultiCamSupported else { // fallback to AVCaptureSession with one camera return } 

When multi-cam session is active, the maximum resolution of each camera is reduced — on iPhone 13 Pro you can get at most 1920×1440 from the back and 1280×960 from the front simultaneously. Attempting to set 4K on both results in AVCaptureSessionRuntimeErrorNotification with code AVError.outOfMemory. In production — we fix 1280×720 for both, which is sufficient for streaming.

Thermal state. Simultaneous operation of two ISPs (Image Signal Processor) and GPU composition heats up the device quickly. Example on iPhone 12 mini: after 30 minutes of streaming with two cameras, thermal throttling triggers, and the system forcibly reduces framerate to 20fps. Solution — monitor ProcessInfo.ThermalState and at .serious switch to one camera or reduce bitrate.

Android with Camera2 API: simultaneous capture is supported via CameraManager.getCameraCharacteristics + LOGICAL_MULTI_CAMERA or explicit opening of two physical cameras. In practice, support depends on the OEM — Samsung Galaxy S22+ delivers two streams, budget Xiaomi may return ERROR_CAMERA_IN_USE. Pre-release testing on target device park is mandatory.

During long broadcasts, quality may degrade; we typically use 720p for both cameras to balance performance and visual fidelity.

Metal Composition Outperforms CIFilter

CIFilter is convenient but slow: each application takes ~3-5 ms per frame on iPhone 12. At 30 FPS, that's 90-150 ms on GPU, leaving little resources for the encoder. Metal shader does composition in ~0.5 ms per frame — 6-10 times faster. Comparison table:

Metal composition is 2x more performant than CIFilter during long broadcasts.

How We Build the Pipeline

Implementation steps:

1. Check device support — verify AVCaptureMultiCamSession.isMultiCamSupported on iOS or logical multi-camera on Android.
2. Configure session — add two AVCaptureDeviceInput instances and two AVCaptureVideoDataOutput queues.
3. Set up Metal composer — create a custom shader that renders back camera full-frame and front camera as PiP rectangle.
4. Implement synchronizer — use AVCaptureDataOutputSynchronizer to align frame timestamps.
5. Encode and stream — feed composed frames to VideoToolbox (H.264/H.265) and transmit via SRT/RTMP using libsrt or HaishinKit.

Key point — composition. Two video streams cannot be directly fed into one encoder. You need to mix frames via MTKView or CIFilter. We use Metal with custom shader: the back camera takes full-frame, the front camera renders as a PiP rectangle in the corner.

import Metal import AVFoundation // Get CMSampleBuffer from each camera on different queues let backQueue = DispatchQueue(label: "back.camera") let frontQueue = DispatchQueue(label: "front.camera") backOutput.setSampleBufferDelegate(self, queue: backQueue) frontOutput.setSampleBufferDelegate(self, queue: frontQueue) // Synchronize via AVCaptureDataOutputSynchronizer let synchronizer = AVCaptureDataOutputSynchronizer( dataOutputs: [backOutput, frontOutput] ) synchronizer.setDelegate(self, queue: syncQueue) 

AVCaptureDataOutputSynchronizer is mandatory. Without it, frames from both cameras arrive with desynchronization up to 33ms (one frame at 30fps), and the PiP window appears jittery relative to the main stream.

For SRT streaming (as a more stable alternative to RTMP on mobile) — we use libsrt compiled for iOS/Android, or HaishinKit 2.x with built-in SRT support.

Managing PiP Position During Stream

We make the PiP window draggable via UIPanGestureRecognizer with snap-to-corners animation. Coordinates are saved in UserDefaults — the user should not reposition each time.

On orientation change, the Metal shader receives new PiP coordinates automatically via CADisplayLink, which recalculates layout on each frame.

Typical Mistakes

• Not adding AVCaptureMultiCamSession to background mode (UIBackgroundModes: audio) — when the app is minimized, iOS will kill the session after 30 seconds.
• Ignoring sessionWasInterrupted on incoming call — need to pause the stream and resume in sessionInterruptionEnded.
• Using DispatchQueue.main for processing CMSampleBuffer — decoding and Metal rendering on the main thread drop UI by 8–12ms per frame.
• Setting too high resolution without considering thermal throttling and bitrate negotiation.

What's Included in the Work

• Architectural documentation describing flows and thermal triggers.
• Source code with comments in Swift/Kotlin.
• Integration with your backend (REST/WebSocket/GraphQL).
• Guidelines for publishing to App Store and Google Play.
• 1 month of support after project delivery.

We guarantee stable operation under load and compliance with Apple and Google guidelines.

Timelines and Cost

iOS implementation with Metal composer, PiP, SRT/RTMP streaming, and thermal testing: 4–6 weeks. Android with Camera2 API — plus 2–3 weeks due to device fragmentation. Cost ranges from $15,000 for iOS to $25,000 for both platforms, excluding travel expenses. Exact pricing is calculated individually after requirements analysis.

Get a consultation on your project — we will assess feasibility and provide a detailed estimate. Contact us to discuss technical specifications and deployment timeline.