AI Image Content Moderation for Mobile App

AI-Powered Image Content Moderation for Mobile Apps

Images are harder to moderate than text. Users try to bypass filters: edit photos, change resolution, add stickers over problematic content. Reliable system is multi-layered: client, server, async check, hash database of known content.

Why Single-Layer Approach Fails

Hash comparison PhotoDNA/CSAM is good for known content detection, not new content. Vision API alone can be bypassed with light image processing. Need combination.

Client-Side Check: Vision Safe Search

On iOS VNClassifyImageRequest has categories like explicit, but insufficient signal. Best on-device option — CoreML model like NudeNet-mobile (open source, ~8 MB):

class LocalImageModerator {
    private let model: NudeNetMobile

    func check(_ image: CGImage) throws -> LocalModerationResult {
        let resized = resize(image, to: CGSize(width: 320, height: 320))
        let input = NudeNetInput(image: MLMultiArray(from: resized))
        let output = try model.prediction(input: input)

        // Classes: SAFE / EXPOSED_BREAST / EXPOSED_GENITALIA / etc.
        let topClass = output.classLabels.max(by: { output.classProbability[$0]! < output.classProbability[$1]! })!
        return LocalModerationResult(
            isSafe: topClass == "SAFE",
            confidence: output.classProbability[topClass]!
        )
    }
}

Inference time — 30–50 ms on iPhone 13. Apply BEFORE uploading to server: if client model blocks — save bandwidth and money on server check.

Server-Side Moderation: AWS Rekognition

AWS Rekognition DetectModerationLabels is industry standard. Good accuracy, supports hierarchical labels:

# Backend
import boto3

rekognition = boto3.client('rekognition', region_name='eu-west-1')

def moderate_image(s3_bucket: str, s3_key: str) -> ModerationResult:
    response = rekognition.detect_moderation_labels(
        Image={'S3Object': {'Bucket': s3_bucket, 'Name': s3_key}},
        MinConfidence=60.0
    )
    labels = response['ModerationLabels']
    top_level = [l for l in labels if not l.get('ParentName')]

    blocked_categories = {'Explicit Nudity', 'Violence', 'Visually Disturbing'}
    for label in top_level:
        if label['Name'] in blocked_categories and label['Confidence'] > 80:
            return ModerationResult(blocked=True, reason=label['Name'],
                                    confidence=label['Confidence'])
    return ModerationResult(blocked=False)

Price: ~$0.001 per image (1000 requests = $1). For apps with active UGC — significant cost; optimize via client pre-check.

Google Cloud Vision Safe Search — alternative with similar pricing. Returns adult, violence, racy, spoof, medical as likelihoods from VERY_UNLIKELY to VERY_LIKELY.

PhotoDNA / Hash-Based CSAM Detection

For apps with public UGC — legal requirement in some jurisdictions. Microsoft PhotoDNA SDK — perceptual hashing, resistant to cropping, scaling, slight compression. Hash compared against known illegal content database (NCMEC Hash Database in USA).

Integration usually via partnership with NCMEC or Microsoft. Not open source. For Russian market — IWF (Internet Watch Foundation) provides similar service.

Simple perceptual hash (pHash) for deduplication — available via open-source:

// Android: pHash via dcperceptualhash
fun computePHash(bitmap: Bitmap): Long {
    val scaled = Bitmap.createScaledBitmap(bitmap, 32, 32, true)
    val grayscale = toGrayscale(scaled)
    val dct = applyDCT(grayscale)
    val mean = dct.average()
    return dct.foldIndexed(0L) { i, acc, v -> if (v > mean) acc or (1L shl i) else acc }
}

// Hamming distance <= 10 = similar images
fun hammingDistance(a: Long, b: Long): Int = java.lang.Long.bitCount(a xor b)

Async Check and Retroactive Deletion

Sync moderation on upload is necessary but insufficient. Add async layer:

1. Image passes sync check → published
2. Async: heavier model (GPT-4 Vision, expensive Rekognition endpoint) re-checks
3. If flagged — content marked for review or auto-deleted

For high-load apps — separate queue SQS/RabbitMQ for async moderation, worker processes.

UX on Block

User must understand why photo rejected and appeal:

// iOS: show rejection screen with appeal button
struct ModerationRejectionView: View {
    let reason: ModerationReason

    var body: some View {
        VStack {
            Image(systemName: "exclamationmark.triangle")
            Text("Photo doesn't meet community guidelines")
            Text(reason.userFriendlyDescription)
                .foregroundStyle(.secondary)
            Button("Appeal") { /* open appeal form */ }
            Button("Choose another photo") { /* dismiss */ }
        }
    }
}

Appeal — form with text field, goes to ticket system for manual review. Respond within 24–48 hours.

Timelines

Backend with Rekognition + client CoreML pre-check — 4–6 days. Full system with pHash, async checking, appeals, analytics — 3–4 weeks. Cost depends on content volume and accuracy requirements.