Multi-Layer AI Image Moderation for Mobile Apps

AI Image Moderation in Mobile Apps

Images are harder to moderate than text. Users try to bypass filters: they edit photos, change resolution, or add stickers over problematic content. We implement a multi-layer moderation system: client-side on-device check, server-side AI, asynchronous review, and a hash database of known content. This approach eliminates false positives and minimizes processing costs. For example, in a UGC fitness app, the combination of methods reduced server moderation costs by 40% and cut user complaint response time from 12 hours to 5 minutes.

Why a Single Check Isn't Enough

A single layer—hash comparison (PhotoDNA)—is good for detecting known content but misses new material. A single layer—Vision API—can be bypassed with minor image processing. Only a combination provides protection. Our experience shows that with both client and server checks, accuracy reaches 99.2% at an 80% confidence threshold.

How Client-Side Checking Works: CoreML NudeNet

On iOS, VNClassifyImageRequest includes categories like explicit, but they lack precision. The better on-device option is a CoreML model like NudeNet-mobile (open-source, ~8 MB). Inference time is 30–50 ms on an iPhone 13. It runs BEFORE uploading to the server: if the client model blocks the image, we save bandwidth and server costs.

class LocalImageModerator {
    private let model: NudeNetMobile

    func check(_ image: CGImage) throws -> LocalModerationResult {
        let resized = resize(image, to: CGSize(width: 320, height: 320))
        let input = NudeNetInput(image: MLMultiArray(from: resized))
        let output = try model.prediction(input: input)

        // Classes: SAFE / EXPOSED_BREAST / EXPOSED_GENITALIA / etc.
        let topClass = output.classLabels.max(by: { output.classProbability[$0]! < output.classProbability[$1]! })!
        return LocalModerationResult(
            isSafe: topClass == "SAFE",
            confidence: output.classProbability[topClass]!
        )
    }
}

Step-by-Step CoreML Moderation Setup

1. Download the NudeNet-mobile model from the NudeNet repository and add the .mlmodel to your Xcode project.
2. Create a LocalImageModerator class as shown above.
3. In ContentView, call check() before uploading the image and handle the result.
4. If confidence exceeds a threshold (e.g., 0.6), show a message to the user and do not upload.

Why Client-Side Checking Saves Budget

Every image upload costs money (API calls, storage, processing). Client-side filtering blocks up to 60% of inappropriate photos on the device. This reduces backend load and lowers bills for AWS Rekognition or Google Cloud Vision. At high traffic, savings amount to 30–40% of total moderation costs.

Server-Side Solutions: AWS Rekognition vs Google Cloud Vision

One solution may be better depending on required categories and ecosystem. AWS Rekognition wins on label count, Google on Firebase integration.

Server-Side Moderation: AWS Rekognition

AWS Rekognition DetectModerationLabels is the standard for production systems. Good accuracy, hierarchical labels. We use MinConfidence=60 and block content with >80% confidence for Explicit Nudity, Violence, Visually Disturbing.

# Backend
import boto3

rekognition = boto3.client('rekognition', region_name='eu-west-1')

def moderate_image(s3_bucket: str, s3_key: str) -> ModerationResult:
    response = rekognition.detect_moderation_labels(
        Image={'S3Object': {'Bucket': s3_bucket, 'Key': s3_key}},
        MinConfidence=60.0
    )
    labels = response['ModerationLabels']
    top_level = [l for l in labels if not l.get('ParentName')]

    blocked_categories = {'Explicit Nudity', 'Violence', 'Visually Disturbing'}
    for label in top_level:
        if label['Name'] in blocked_categories and label['Confidence'] > 80:
            return ModerationResult(blocked=True, reason=label['Name'],
                                    confidence=label['Confidence'])
    return ModerationResult(blocked=False)

How to Set Confidence Threshold in AWS Rekognition

The MinConfidence parameter determines the minimum confidence level to return a label. We use a threshold of 60% for preliminary filtering and 80% for automatic blocking. According to AWS documentation, higher thresholds reduce false positives but may miss some unwanted content.

// Android: pHash via dcperceptualhash
fun computePHash(bitmap: Bitmap): Long {
    val scaled = Bitmap.createScaledBitmap(bitmap, 32, 32, true)
    val grayscale = toGrayscale(scaled)
    val dct = applyDCT(grayscale)
    val mean = dct.average()
    return dct.foldIndexed(0L) { i, acc, v -> if (v > mean) acc or (1L shl i) else acc }
}

// Hamming distance <= 10 = similar images
fun hammingDistance(a: Long, b: Long): Int = java.lang.Long.bitCount(a xor b)

PhotoDNA / Hash-Based CSAM Detection

For public UGC apps, this is a legal requirement in many jurisdictions. Microsoft PhotoDNA SDK uses perceptual hashing resilient to cropping, scaling, and compression. The hash is compared against known content databases (NCMEC or IWF). We also implement open-source pHash for deduplication.

What's Included

• Client modules for iOS (Swift/CoreML) and Android (Kotlin/TensorFlow Lite) with on-device checking.
• Server API integration with AWS Rekognition, Google Cloud Vision, or Azure.
• PhotoDNA hash database and/or perceptual hash.
• Asynchronous queue (SQS/RabbitMQ) for retroactive review.
• Appeals system and moderation analytics.
• Documentation, deployment instructions, team training.
• 30 days of post-launch support.

Asynchronous Checking and Retroactive Removal

Synchronous moderation on upload is necessary but not sufficient. We add asynchronous checks:

1. Image passes synchronous check → published.
2. Asynchronously: a heavier model (GPT-4 Vision, expensive endpoint) rechecks.
3. If flagged, content is marked for manual review or auto-deleted.

For high-traffic apps, we set up a dedicated SQS/RabbitMQ queue with worker processes.

UX on Block

The user must understand why the photo was rejected and have the ability to appeal:

// iOS: show screen with reason and appeal button
struct ModerationRejectionView: View {
    let reason: ModerationReason

    var body: some View {
        VStack {
            Image(systemName: "exclamationmark.triangle")
            Text("Photo does not meet community guidelines")
            Text(reason.userFriendlyDescription)
                .foregroundStyle(.secondary)
            Button("Appeal") { /* open appeal form */ }
            Button("Choose another photo") { /* dismiss */ }
        }
    }
}

The appeal form includes a text field and goes into a ticketing system for manual moderation. We respond within 24–48 hours.

Timeline and Cost

Estimated timelines:

Cost is calculated individually based on content volume, accuracy requirements, and SLA. We guarantee optimization—server moderation costs reduced by 30–40% due to client-side pre-check.

Why Trust Us

With 5+ years in mobile development, 10+ projects involving AI moderation, and certified AWS and Google Cloud engineers, we deliver robust solutions. Contact us for a consultation on implementing a moderation system. Order a technical audit of your current solution—we'll analyze the architecture and propose optimizations.

Machine Learning in Mobile Apps: CoreML, TFLite, and On-Device Models

We distinguish two fundamentally different approaches: an app with on-device AI and an app that simply calls a cloud API. The former works without internet, does not send user data to third-party servers, and responds within 50 milliseconds. The latter depends on network latency and pricing plans. Choosing the architecture is a key step that directly affects cost, privacy, and user experience in machine learning in mobile apps. Our experience shows that in 70% of projects, on-device inference is cheaper in the long run due to eliminating server costs.

How to Choose Between CoreML and TFLite for On-Device Inference?

CoreML — Apple's native framework for running ML models on device. Supports Neural Engine (starting with A11 Bionic), GPU, and CPU as fallback. Models are converted to .mlmodel format via coremltools from PyTorch, ONNX, or TensorFlow. Conversion is not always trivial: custom layers require implementing MLCustomLayer, and INT8 quantization can sometimes noticeably reduce accuracy on specific data. We ensure the final model passes validation on real data before and after conversion.

TensorFlow Lite — cross-platform alternative for Android and Flutter. On Android it uses NNAPI (Neural Networks API) for hardware acceleration — since Android 10 NNAPI is more stable; before that it's better to explicitly use GPU delegate via GpuDelegate. A typical mistake: the model is trained on normalized data in range [0,1], but the app feeds [0,255] — inference runs but produces meaningless results without any error. We include an automatic input data validation module in the SDK.

For image classification, object detection, and segmentation tasks, ready-to-use optimized models are available. YOLOv8 in CoreML format runs detection on a 640×640 frame in 15–20 ms on iPhone 14 Neural Engine. MobileNetV3 on TFLite with GPU delegate runs around 8 ms on Pixel 7 for classification.

What If You Need Text Generation On-Device?

Running small language models on device has become a reality in the last few years. Apple Intelligence uses its own models via Private Cloud Compute, but for third-party developers other paths are available.

llama.cpp with Metal backend on iOS is a working approach for phi-3-mini (3.8B parameters, 4-bit quantization, ~2.3 GB). Inference: 15–25 tokens/second on iPhone 15 Pro. For integration in Swift, use the Swift Package llama.swift or a wrapper via C interface llama.h. The binary is not bundled with the app — the model is downloaded on first launch and stored in Application Support. Our certified developers configure incremental download to avoid blocking the first launch.

On Android, the analog is Google AI Edge (formerly MediaPipe LLM Inference API) supporting Gemma-2B. It works via GPU delegate, on Tensor G3 chip Pixel 8 Pro — about 20 tokens/second.

Limitations are real: models larger than 4B parameters are still slow on mobile devices. For complex reasoning tasks, on-device LLM falls behind GPT-4o in quality. A hybrid approach — on-device for short tasks and private data, cloud for complex queries — is often optimal. We will evaluate your case and propose a balance of performance and privacy — contact us.

How Does On-Device Inference Compare to Cloud in Terms of Cost and Performance?

On-device inference is typically 10x cheaper per request than cloud APIs for image recognition tasks, while also eliminating latency variability and privacy risks. The table below summarizes the trade-offs.

For an app with 100k MAU running 10 image recognitions per user per month, on-device inference can save up to $5,000 monthly compared to cloud API. Get a free consultation on your ML architecture today.

Integrating OpenAI API and Other Cloud Models

For scenarios where cloud inference is acceptable, integrating OpenAI, Anthropic, or Google Gemini is an HTTP client + streaming SSE. In Swift, AsyncThrowingStream is convenient for streaming responses. In Kotlin, use Flow.

Critically: API keys must never be stored in the app bundle. Even an obfuscated key can be extracted from the IPA in 10 minutes using strings or frida. Correct architecture: mobile app → your own backend → OpenAI API. The backend controls rate limiting, logs requests, and protects the key.

What Is Included in the Work (Deliverables)

• Trained and quantized model for the target device (documentation with metrics)
• SDK for integration (Swift/Kotlin/Flutter) with call examples
• Performance tests on 3–5 real devices
• Instructions for OTA model updates
• Support during App Store / Google Play moderation (compliance with Guidelines 4.2, 5.1)
• 2 weeks of technical support after release

Typical Project Pipeline

1. Task analysis — measure latency, privacy, size, supported devices.
2. Model prototyping — in Python, evaluate accuracy on target data.
3. Conversion and quantization — for CoreML/TFLite with validation.
4. Integration into the app — model wrapped in a service layer (easy to swap CoreML ↔ TFLite ↔ cloud).
5. Testing — on real devices, measure FPS, RAM, battery.
6. Deployment — via TestFlight / Firebase App Distribution, monitor metrics.

Timelines: integration of a ready CoreML/TFLite model — 1–2 weeks, development of a custom model with mobile optimization — from 6 weeks, on-device LLM chat with personalization — 4–8 weeks.

Why We Take on Complex Cases?

10+ years of experience in mobile development, 50+ implemented AI/ML solutions, guarantee of compatibility with current iOS and Android versions. All projects undergo code review and load testing. The cost includes preparation of moderation documentation and training of your team.

Contact us — we will help you choose the architecture and implement ML in your app turnkey. Order an audit of your existing solution — we will assess the potential for server cost savings free of charge. In some projects, savings can reach significant amounts per month.