Implementing AI Video Anomaly Detection in Mobile Apps
We often get this request: "I want to detect falls, intrusions, or abnormal behavior on video, but I don't know where to start." The challenge is that anomalies are rare events without clear labels. It's impossible to enumerate all variations of "bad" behavior in advance. That's why we use a two-tier approach: fast deterministic rules for known scenarios and unsupervised AI for subtle anomalies. This balances speed and coverage: rules fire in milliseconds, while the AI adapts to your data.
On one project, a client wanted to monitor server room intrusions and also flag suspicious employee behavior, like unusually frequent passes outside working hours. Deterministic rules easily caught zone violations, and the autoencoder revealed atypical routes that had previously gone unnoticed. The result: 95% detection accuracy with zero false positives on normal situations.
How to Distinguish Spatial from Behavioral Anomalies?
Before writing code, we precisely define requirements with the client. Types of anomalies:
- Spatial – object in a zone where it shouldn't be (person in server room)
- Behavioral – normal object behaves unusually (running where others walk)
- Temporal – event occurs at the wrong time (movement at night)
- Technical – equipment malfunctions (smoke, vibration)
Each type requires a different architecture. For spatial anomalies, deterministic rules suffice; for behavioral, an autoencoder is needed.
How We Build the Two-Tier System
We split detection into two stages. First, fast rules based on computer vision (person detector and zones). Second, an AI model that runs only when no rule violations are present, saving resources. This approach is 3 times more CPU-efficient than using one heavy model for everything.
Tier 1: Deterministic Rules
Rules are set in configuration: forbidden zone coordinates, schedules, object types. They work without training, instantly and predictably.
class AnomalyRulesEngine {
struct RestrictedZone {
let polygon: [CGPoint] // normalized coordinates
let schedule: WorkSchedule? // nil = always restricted
let name: String
}
private let restrictedZones: [RestrictedZone]
private let personDetector: VNCoreMLModel // lightweight YOLOv8n
func check(frame: CVPixelBuffer, timestamp: Date) -> [RuleViolation] {
let persons = detectPersons(frame)
var violations: [RuleViolation] = []
for person in persons {
let personCenter = person.boundingBox.center
for zone in restrictedZones {
if zone.polygon.contains(personCenter) {
if let schedule = zone.schedule, !schedule.isActive(at: timestamp) {
violations.append(RuleViolation(
type: .unauthorizedZoneAccess,
zone: zone.name,
timestamp: timestamp
))
} else if zone.schedule == nil {
violations.append(RuleViolation(type: .restrictedZone, zone: zone.name))
}
}
}
}
return violations
}
}
Tier 2: Autoencoder for Non-Obvious Anomalies
For behavioral anomalies, we use an autoencoder: trained on normal behavior, anomaly = high reconstruction error. The autoencoder is 10 times lighter than a YOLO detector (5 MB vs 50 MB), critical for mobile devices.
# Training autoencoder on normal video fragments
import torch
import torch.nn as nn
class VideoAnomalyAutoencoder(nn.Module):
"""
Input tensor: [batch, frames, height, width, channels]
Trained only on NORMAL scenes
Anomaly: reconstruction_error > threshold
"""
def __init__(self, input_shape=(16, 64, 64, 3)):
super().__init__()
self.encoder = nn.Sequential(
nn.Conv3d(3, 32, kernel_size=(3,3,3), padding=1),
nn.ReLU(),
nn.MaxPool3d((1,2,2)),
nn.Conv3d(32, 64, kernel_size=(3,3,3), padding=1),
nn.ReLU(),
nn.MaxPool3d((2,2,2)),
)
self.decoder = nn.Sequential(
nn.ConvTranspose3d(64, 32, kernel_size=(3,3,3),
stride=(2,2,2), padding=1, output_padding=1),
nn.ReLU(),
nn.ConvTranspose3d(32, 3, kernel_size=(3,3,3),
stride=(1,2,2), padding=1, output_padding=(0,1,1)),
nn.Sigmoid()
)
def forward(self, x):
z = self.encoder(x)
return self.decoder(z)
def anomaly_score(self, x):
reconstructed = self(x)
return ((x - reconstructed) ** 2).mean(dim=[1,2,3,4])
The anomaly threshold is set at the 99th percentile of reconstruction error on the normal dataset. On mobile, this autoencoder is converted to CoreML or TFLite.
Mobile Inference: Sliding Window Processing
Processing each window of 16 frames takes 30-50 ms on an iPhone 12, enabling 20-30 fps.
// iOS: analyzing video stream with sliding window of 16 frames
class SlidingWindowAnalyzer {
private var frameBuffer: CircularBuffer<CVPixelBuffer> = CircularBuffer(capacity: 16)
private var frameCounter = 0
private let stepSize = 8 // new window every 8 frames (50% overlap)
func addFrame(_ frame: CVPixelBuffer) async -> AnomalyScore? {
frameBuffer.append(frame)
frameCounter += 1
guard frameCounter % stepSize == 0,
frameBuffer.count == 16 else { return nil }
return try? await computeAnomalyScore(frames: Array(frameBuffer))
}
private func computeAnomalyScore(frames: [CVPixelBuffer]) async throws -> AnomalyScore {
let tensor = prepareTensor(frames) // [1, 16, 64, 64, 3]
let output = try autoencoderModel.prediction(input: tensor)
let score = output.anomalyScore.floatValue
return AnomalyScore(
value: score,
isAnomaly: score > anomalyThreshold,
frameWindow: frames
)
}
}
Alerts and Reaction
We implement a multi-level system: warnings and critical alerts with cooldown to avoid spam. Critical alerts go via webhook to an external security system.
// Android: multi-level alert system
sealed class AnomalyAlert {
data class Warning(val message: String, val score: Float) : AnomalyAlert()
data class Critical(val message: String, val violations: List<RuleViolation>) : AnomalyAlert()
}
class AlertManager(private val notificationManager: NotificationManager) {
private val cooldownMap = mutableMapOf<String, Long>()
private val alertCooldownMs = 30_000L // no spam: max once per 30 sec
fun emit(alert: AnomalyAlert, alertKey: String) {
val lastAlertTime = cooldownMap[alertKey] ?: 0L
if (System.currentTimeMillis() - lastAlertTime < alertCooldownMs) return
cooldownMap[alertKey] = System.currentTimeMillis()
when (alert) {
is AnomalyAlert.Warning -> showLocalNotification(alert.message, priority = LOW)
is AnomalyAlert.Critical -> {
showLocalNotification(alert.message, priority = HIGH)
sendWebhook(alert) // integration with external security system
}
}
}
}
Comparison of Approaches
| Feature | Deterministic Rules | Autoencoder AI |
|---|---|---|
| Anomaly type | Spatial, temporal | Behavioral, technical |
| Training | Expert-defined zones/schedule | Unsupervised on normal data |
| Accuracy | 100% for defined rules | Data-dependent (ROC-AUC ~0.9) |
| Model size | 0 (lightweight face detector) | 5-15 MB |
| Latency on iOS | <5 ms | 30-50 ms per window |
Deterministic rules are 10 times faster than AI, but miss new anomaly types. The autoencoder finds what wasn't predefined — perfect as a second tier.
What Our Work Includes
Our team (8+ years in mobile CV, 15+ projects) offers:
- Collect and label normal data for training
- Design deterministic rules for your scenario
- Train and convert autoencoder to CoreML / TFLite
- Integrate with iOS (Swift 5.9+, SwiftUI) and Android (Kotlin, Jetpack Compose)
- Multi-level alert system with webhook integration
- Documentation and operator training
- Post-deployment support
We'll assess your project within 2 days. Contact us for a consultation — we'll show a demo on real data. Request a pilot launch and see the effectiveness.
Timeline Estimates
| Stage | Duration |
|---|---|
| Zone violation detection (without AI) | 1-2 weeks |
| Full system with autoencoder, sliding window, alerts | 2-4 weeks |
| Collect normal data and train model | +1-2 weeks |
| Integration with external security system | +1 week |
Cost is calculated individually. We guarantee code quality and full documentation. We use CoreML and TFLite — proven tools for mobile inference.