Implementing AI Video Anomaly Detection in Mobile Apps

We often get this request: "I want to detect falls, intrusions, or abnormal behavior on video, but I don't know where to start." The challenge is that anomalies are rare events without clear labels. It's impossible to enumerate all variations of "bad" behavior in advance. That's why we use a two-tier approach: fast deterministic rules for known scenarios and unsupervised AI for subtle anomalies. This balances speed and coverage: rules fire in milliseconds, while the AI adapts to your data.

On one project, a client wanted to monitor server room intrusions and also flag suspicious employee behavior, like unusually frequent passes outside working hours. Deterministic rules easily caught zone violations, and the autoencoder revealed atypical routes that had previously gone unnoticed. The result: 95% detection accuracy with zero false positives on normal situations.

How to Distinguish Spatial from Behavioral Anomalies?

Before writing code, we precisely define requirements with the client. Types of anomalies:

• Spatial – object in a zone where it shouldn't be (person in server room)
• Behavioral – normal object behaves unusually (running where others walk)
• Temporal – event occurs at the wrong time (movement at night)
• Technical – equipment malfunctions (smoke, vibration)

Each type requires a different architecture. For spatial anomalies, deterministic rules suffice; for behavioral, an autoencoder is needed.

How We Build the Two-Tier System

We split detection into two stages. First, fast rules based on computer vision (person detector and zones). Second, an AI model that runs only when no rule violations are present, saving resources. This approach is 3 times more CPU-efficient than using one heavy model for everything.

Tier 1: Deterministic Rules

Rules are set in configuration: forbidden zone coordinates, schedules, object types. They work without training, instantly and predictably.

class AnomalyRulesEngine { struct RestrictedZone { let polygon: [CGPoint] // normalized coordinates let schedule: WorkSchedule? // nil = always restricted let name: String } private let restrictedZones: [RestrictedZone] private let personDetector: VNCoreMLModel // lightweight YOLOv8n func check(frame: CVPixelBuffer, timestamp: Date) -> [RuleViolation] { let persons = detectPersons(frame) var violations: [RuleViolation] = [] for person in persons { let personCenter = person.boundingBox.center for zone in restrictedZones { if zone.polygon.contains(personCenter) { if let schedule = zone.schedule, !schedule.isActive(at: timestamp) { violations.append(RuleViolation( type: .unauthorizedZoneAccess, zone: zone.name, timestamp: timestamp )) } else if zone.schedule == nil { violations.append(RuleViolation(type: .restrictedZone, zone: zone.name)) } } } } return violations } } 

Tier 2: Autoencoder for Non-Obvious Anomalies

For behavioral anomalies, we use an autoencoder: trained on normal behavior, anomaly = high reconstruction error. The autoencoder is 10 times lighter than a YOLO detector (5 MB vs 50 MB), critical for mobile devices.

# Training autoencoder on normal video fragments import torch import torch.nn as nn class VideoAnomalyAutoencoder(nn.Module): """ Input tensor: [batch, frames, height, width, channels] Trained only on NORMAL scenes Anomaly: reconstruction_error > threshold """ def __init__(self, input_shape=(16, 64, 64, 3)): super().__init__() self.encoder = nn.Sequential( nn.Conv3d(3, 32, kernel_size=(3,3,3), padding=1), nn.ReLU(), nn.MaxPool3d((1,2,2)), nn.Conv3d(32, 64, kernel_size=(3,3,3), padding=1), nn.ReLU(), nn.MaxPool3d((2,2,2)), ) self.decoder = nn.Sequential( nn.ConvTranspose3d(64, 32, kernel_size=(3,3,3), stride=(2,2,2), padding=1, output_padding=1), nn.ReLU(), nn.ConvTranspose3d(32, 3, kernel_size=(3,3,3), stride=(1,2,2), padding=1, output_padding=(0,1,1)), nn.Sigmoid() ) def forward(self, x): z = self.encoder(x) return self.decoder(z) def anomaly_score(self, x): reconstructed = self(x) return ((x - reconstructed) ** 2).mean(dim=[1,2,3,4]) 

The anomaly threshold is set at the 99th percentile of reconstruction error on the normal dataset. On mobile, this autoencoder is converted to CoreML or TFLite.

Mobile Inference: Sliding Window Processing

Processing each window of 16 frames takes 30-50 ms on an iPhone 12, enabling 20-30 fps.

// iOS: analyzing video stream with sliding window of 16 frames class SlidingWindowAnalyzer { private var frameBuffer: CircularBuffer<CVPixelBuffer> = CircularBuffer(capacity: 16) private var frameCounter = 0 private let stepSize = 8 // new window every 8 frames (50% overlap) func addFrame(_ frame: CVPixelBuffer) async -> AnomalyScore? { frameBuffer.append(frame) frameCounter += 1 guard frameCounter % stepSize == 0, frameBuffer.count == 16 else { return nil } return try? await computeAnomalyScore(frames: Array(frameBuffer)) } private func computeAnomalyScore(frames: [CVPixelBuffer]) async throws -> AnomalyScore { let tensor = prepareTensor(frames) // [1, 16, 64, 64, 3] let output = try autoencoderModel.prediction(input: tensor) let score = output.anomalyScore.floatValue return AnomalyScore( value: score, isAnomaly: score > anomalyThreshold, frameWindow: frames ) } } 

Alerts and Reaction

We implement a multi-level system: warnings and critical alerts with cooldown to avoid spam. Critical alerts go via webhook to an external security system.

// Android: multi-level alert system sealed class AnomalyAlert { data class Warning(val message: String, val score: Float) : AnomalyAlert() data class Critical(val message: String, val violations: List<RuleViolation>) : AnomalyAlert() } class AlertManager(private val notificationManager: NotificationManager) { private val cooldownMap = mutableMapOf<String, Long>() private val alertCooldownMs = 30_000L // no spam: max once per 30 sec fun emit(alert: AnomalyAlert, alertKey: String) { val lastAlertTime = cooldownMap[alertKey] ?: 0L if (System.currentTimeMillis() - lastAlertTime < alertCooldownMs) return cooldownMap[alertKey] = System.currentTimeMillis() when (alert) { is AnomalyAlert.Warning -> showLocalNotification(alert.message, priority = LOW) is AnomalyAlert.Critical -> { showLocalNotification(alert.message, priority = HIGH) sendWebhook(alert) // integration with external security system } } } } 

Comparison of Approaches

Deterministic rules are 10 times faster than AI, but miss new anomaly types. The autoencoder finds what wasn't predefined — perfect as a second tier.

What Our Work Includes

Our team (8+ years in mobile CV, 15+ projects) offers:

• Collect and label normal data for training
• Design deterministic rules for your scenario
• Train and convert autoencoder to CoreML / TFLite
• Integrate with iOS (Swift 5.9+, SwiftUI) and Android (Kotlin, Jetpack Compose)
• Multi-level alert system with webhook integration
• Documentation and operator training
• Post-deployment support

We'll assess your project within 2 days. Contact us for a consultation — we'll show a demo on real data. Request a pilot launch and see the effectiveness.

Timeline Estimates

Cost is calculated individually. We guarantee code quality and full documentation. We use CoreML and TFLite — proven tools for mobile inference.