Implementing Video Filter Overlays in Mobile Applications
Photo filters and video filters are different by complexity. A photo is one frame, processable in 50 ms. Video at 30 fps — 30 frames per second, each must be processed faster than 33 ms. Otherwise playback stutters.
Two Modes: Preview and Export
Real-time preview (during playback). Requires GPU processing without writing results to file.
Export (final write with applied filter). May take seconds, but results must be frame-accurate.
iOS: AVVideoComposition + Core Image
AVVideoCompositionCoreAnimationTool for static overlays — suitable for text and logos. For pixel filters — AVVideoComposition with custom AVVideoCompositing:
class FilterCompositor: NSObject, AVVideoCompositing {
func startRequest(_ asyncVideoCompositionRequest: AVAsynchronousVideoCompositionRequest) {
guard let frame = asyncVideoCompositionRequest.sourceFrame(byTrackID: trackID) else { return }
let ciImage = CIImage(cvPixelBuffer: frame)
let filtered = applyFilter(ciImage)
let output = asyncVideoCompositionRequest.renderContext.newPixelBuffer()!
ciContext.render(filtered, to: output)
asyncVideoCompositionRequest.finish(withComposedVideoFrame: output)
}
}
CIContext with Metal backend is mandatory — CIContext(mtlDevice: MTLCreateSystemDefaultDevice()!). Without explicit specification, some devices use CPU rendering and video freezes.
For LUT filters — CIColorCubeWithColorSpace with 64×64×64 table. One CIContext per application — its creation is expensive.
Android: GPUImage and media3 Effects
Real-time preview — GPUImageView with custom GPUImageFilter. Shader written in GLSL:
precision mediump float;
uniform sampler2D inputImageTexture;
varying vec2 textureCoordinate;
void main() {
vec4 color = texture2D(inputImageTexture, textureCoordinate);
float luma = dot(color.rgb, vec3(0.299, 0.587, 0.114));
gl_FragColor = vec4(mix(vec3(luma), color.rgb, 1.3), color.a);
}
For export with filter — media3 Transformer with GlEffect. MatrixTextureProcessor accepts shader and applies it to each frame during re-encoding.
Flutter. Full real-time video filters require platform code. Native plugins via MethodChannel — the only working approach for production applications. Packages like video_editor (pub.dev) provide UI, but filtering via FFmpeg on CPU — slow on older devices.
Typical Export Mistakes
Creating a new CIContext per frame — death of performance. On iPhone 13 this gives 3–4 fps instead of 30. Context is created once and reused.
CVPixelBuffer from renderContext.newPixelBuffer() must be returned immediately after finish(withComposedVideoFrame:). Holding the reference causes buffer pool leak and crash with kCVReturnInvalidArgument.
On Android: GPUImageFilter is not thread-safe — cannot apply to one instance from multiple threads simultaneously.
Applying Filter to Live Camera Recording
Even more demanding — filter during recording. On iOS: AVCaptureSession + AVCaptureVideoDataOutput → SampleBufferDelegate → Metal shader → render in MTKView. Chain latency must be < 16 ms for 60fps. kCVPixelBufferPixelFormatTypeKey: kCVPixelFormatType_420YpCbCr8BiPlanarFullRange — YUV format from camera converts to Metal texture faster than BGRA.
On Android: CameraX Analysis + ImageAnalysis.Analyzer → GLES shader → GLSurfaceView. ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST — discard old frames if GPU hasn't processed the previous one.
Timeline
5–7 business days: develop shaders for 5–8 filters, integrate preview and export on iOS and Android. Flutter version with native code — plus 2 days.