Implementing Content Safety Filters for AI Generation in Mobile App
When mobile app generates text, images, or audio via AI, users eventually try getting unwanted content — intentionally or accidentally. Moderation via system prompt ("don't generate harmful content") works worse than it seems: prompt can be bypassed, and you're liable.
What and How to Filter
Text generation. OpenAI Moderation API — free endpoint, returns scores per category: hate, harassment, self-harm, sexual, violence with subcategories. Latency 100–200ms, acceptable as post-filter.
Apply to user input (input moderation) and model response (output moderation). Double-check adds ~200–400ms total latency, but provides protection both layers.
Azure Content Safety — more detailed gradation (safe / low / medium / high severity) and additional categories for regulated markets. Needed if app operates in EU/US with compliance requirements.
Images. DALL-E 3 and Stable Diffusion have built-in safety checkers, but adversarial prompts bypass them. Additional layer — Google Cloud Vision SafeSearch or AWS Rekognition for post-checking generated image.
User Content and UGC Risks
If user uploads content (photo, text) passed to LLM context — separate risk vector. Image may contain embedded text with instructions (prompt injection via OCR), text document — attempt overriding system prompt.
For UGC: moderate before content enters database; moderate when passing into AI pipeline. Don't cache moderation results long — user can change content.
Logging Violations and Appeals
Log each blocked request with violation category, but without full message text (GDPR). Show user understandable message, not technical error code. Provide mechanism disputing false positives — all filters have false positive rate.
Timeline Estimates
Basic OpenAI Moderation API integration — 1 day. Two-layer filtering (input + output) with error handling — 2–3 days. Extended system with logging, metrics, appeal mechanism — 4–5 days.