AI-Powered Moderation for User-Generated Content

Imagine: a forum with 50,000 comments daily. Manual moderation requires 12 people, each misses up to 3% of violations. We encountered this task while modernizing a large marketplace—and solved it with a cascade of ML models. In this article, we'll break down how to build an automatic moderation system that filters spam, toxicity, and unwanted images with >99% accuracy at <50 ms latency.

User-generated content—comments, reviews, images, chat messages—requires constant oversight. Automation becomes essential when volumes exceed thousands of posts per hour. Purely manual moderation doesn't scale: with 10,000 posts per day, a team of five moderators can't physically cope. AI moderation handles the task quickly and cheaply, leaving only edge cases for humans. Guarantee a reduction of manual work by 80% through automation. Beyond toxicity, AI can also perform sentiment analysis to gauge emotional tone, enhancing community engagement.

What Can Be Automatically Moderated

• Text content: spam, profanity, hate speech, threats, personal data in plain sight.
• Images: explicit content, violence, copyright violations (via perceptual hashing).
• Links: phishing, malicious domains.
• Tone: toxic comments without explicit banned words.

Each category requires a separate model or API endpoint—there is no universal solution.

How AI Moderation Works

Architecture relies on one of three patterns.

Synchronous check before publication—the user submits content, the server checks it before saving. Latency 200–800 ms. Suitable for critical scenarios: paid reviews, legally significant publications.

Async queue—content is saved with a pending status, a background worker checks via a queue (RabbitMQ, SQS, Redis Streams). Publication occurs after approval or after N minutes if no violations. Suitable for high-load forums and chats.

Hybrid scheme—fast sync check with simple rules (stop words, length, patterns) + async ML check for content that passes the primary filter.

POST /api/comment
  → sync: banned words check (< 5ms)
  → sync: OpenAI Moderation API (< 300ms)
  → save with status=published/flagged
  → async: image scan if attachments

Which API to Choose?

Comparison of popular tools for text and image moderation.

import openai

def moderate_text(content: str) -> dict:
    response = openai.moderations.create(input=content)
    result = response.results[0]

    if result.flagged:
        categories = {k: v for k, v in result.categories.__dict__.items() if v}
        return {"allowed": False, "categories": categories}

    return {"allowed": True}

Custom Model via Fine-Tuning

For specialized content (professional forum with technical jargon, medical platform), third-party APIs cause many false positives. The solution is fine-tuning on your own data.

Process: collect a dataset of 2000-5000 labeled examples (approved/rejected), fine-tune distilbert-base-multilingual-cased via Hugging Face Transformers, deploy as a separate service.

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="./moderation-model",
    device=0  # GPU
)

def classify_content(text: str) -> tuple[str, float]:
    result = classifier(text, truncation=True, max_length=512)[0]
    return result["label"], result["score"]

Inference on CPU: ~50 ms per text up to 512 tokens. On GPU (T4): ~5 ms.

Image Processing

Before sending to an API, preprocess: resize to 2048px on the longest side, convert to JPEG at quality 85%, strip EXIF metadata. This reduces cost and speeds up response. Automation cuts the moderation team size several times, yielding significant budget savings. Pilot deployment typically pays off within months.

For protection against re-uploading known prohibited content—PhotoDNA (Microsoft) or pHash matching against a hash database. PhotoDNA integrates via Azure; pHash can be implemented independently:

import imagehash
from PIL import Image

def compute_phash(image_path: str) -> str:
    img = Image.open(image_path)
    return str(imagehash.phash(img))

def is_known_violation(phash: str, banned_hashes: set, threshold: int = 10) -> bool:
    for banned in banned_hashes:
        if imagehash.hex_to_hash(phash) - imagehash.hex_to_hash(banned) < threshold:
            return True
    return False

Manual Moderation Dashboard

Automation does not decide on borderline cases—they must be shown to a moderator. The manual moderation queue includes:

• content with confidence 0.4–0.7 (uncertain result);
• content reported by users;
• content from new accounts with no history.

Interface: a list with filters, hotkeys for quick decisions (approve/reject/escalate), decision history linked to the operator, accuracy metrics per operator.

Feedback Loop and Retraining

Models degrade if content patterns shift. Improvement cycle:

1. Save all decisions (automatic and manual) with labels.
2. Weekly analyze discrepancies: where automation was wrong, moderator corrected.
3. Monthly retrain the model on accumulated corrections.
4. A/B test the new version on 10% of traffic before full switch.

Monitoring

Metrics for Grafana/Datadog:

• moderation.requests.total — total volume;
• moderation.latency.p99 — 99th percentile latency;
• moderation.flagged.rate — share of blocked content;
• moderation.false_positive.rate — share of incorrect blocks (by appeals);
• moderation.queue.depth — depth of the manual moderation queue.

Alert: if false_positive.rate > 5% in 24 hours, the model needs review.

What's Included in the Work

• Integration of one or more APIs (OpenAI, Perspective, Rekognition, Azure).
• Development of synchronous middleware or async queue on your stack.
• Dashboard for manual moderation with history and metrics.
• API and architecture documentation.
• Team training (2 hours online).
• 1 month of support post-launch.

Timeline Estimates

Basic integration with OpenAI Moderation API and manual review queue: 2 weeks. Full system with custom model, monitoring, and dashboard: 5-6 weeks. Contact us for a pilot project—we'll assess your load and propose a solution. Request a consultation on integration.

Our team has over 5 years of experience in AI moderation and has completed 50+ projects, helping platforms save up to $50,000 annually by automating their moderation workflow.

AI Integration: Chatbots, RAG, Semantic Search, Recommendations

In 8 out of 10 projects, an "AI chatbot" turns out to be an expensive wrapper over GPT-4o with a system prompt. Without access to real company data. The user asks "how much does the Premium plan cost?" — the bot hallucinates a price out of thin air. Asks "when will my order arrive?" — gets a polite "contact support." This is not integration — it's imitation. We have implemented RAG solutions in 30+ projects over 5 years: from e-commerce stores to medical portals. We guarantee: useful AI assistance begins where the model reads your documents, not generic answers.

How do we build RAG systems?

Retrieval-Augmented Generation — standard architecture: query → find relevant fragments in a vector DB → insert found context → model response. But the devil is in the implementation details. Let's break down key components that determine quality.

Chunking. Cutting a document into 500-token pieces without regard for structure is a guarantee of losing meaning. If the cut lands in the middle of a paragraph, context breaks. Solution — recursive RecursiveCharacterTextSplitter with 10–15% overlap for documentation. For contracts and instructions, we use a semantic splitter: extract headings, lists, code blocks — each section becomes an independent chunk. Difference in search quality: on a medical project, precision increased from 0.55 to 0.84 just by proper chunking.

Embedding model. For Russian-language texts, intfloat/multilingual-e5-large gives a noticeable accuracy boost over outdated text-embedding-ada-002. In our measurements, NDCG@10 on a test set of 10,000 query-document pairs is 12% higher. OpenAI text-embedding-3-large is good for English content, but for Russian we recommend BAAI/bge-m3 or the mentioned e5-large.

Vector DB. If you already have PostgreSQL — pgvector saves resources. Install extension CREATE EXTENSION vector, add column vector(1024), create HNSW index. On a project with 80,000 support articles, p95 search time was 12 ms. That's enough. For catalogs with millions of items — Qdrant or Weaviate: native hybrid search and sharding out of the box.

What does hybrid search give?

Vector-only search is blind to exact matches: SKUs like "ABC-123", proper names, abbreviations are lost. Full-text-only search doesn't catch synonyms and paraphrasing. Combining via RRF (Reciprocal Rank Fusion) gives the best of both worlds: BM25 + vector search, results merged. In practice, recall@20 increases from 0.65 to 0.92 — the difference is noticeable to the user.

Reranking — final filter: top-20 candidates from hybrid search are run through a cross-encoder cross-encoder/ms-marco-MiniLM-L-6-v2. It adds 50–100 ms to response time, but relevance improves by another 5–10%. Without reranking, the chatbot may show irrelevant documents.

How to implement semantic search on a site?

A search for "comfortable leather armchairs" should find products described as "soft chairs made of natural leather" — ordinary LIKE search cannot do this. Our architecture: when adding a product/post, automatically generate an embedding via multilingual-e5-large, store it in pgvector. On query, embed it with the same model, search nearest neighbors via cosine distance with HNSW index. For a catalog of 100,000 items, index builds in 3 minutes, memory ~400 MB (1536-dimensional vectors). Average search time: 20 ms.

What about recommendation systems?

Collaborative filtering ("users like you bought X") requires history — at least 2–3 months of data with 1000+ active users. For startups or small projects, we use content-based: embedding of current product → search nearest neighbors by cosine similarity. When enough statistics accumulate (usually 15–20 interactions per user), we switch to a hybrid LightFM model. It combines behavior and product features. In our e-commerce project with 50,000 SKUs, the hybrid model increased conversion in the recommendation block by 18% (A/B test lasted 2 weeks).

How does streaming work?

Users shouldn't wait for the entire text to be generated — it kills UX. Server-Sent Events (SSE) is the protocol for token streaming. OpenAI SDK supports stream: true, returning an AsyncIterator. On frontend — Vercel AI SDK (useChat) or custom EventSource. Typical mistake: using WebSocket for unidirectional streaming — SSE is simpler (less code, built-in reconnect). Stack: Node.js + SSE + React.

How to orchestrate agents?

A simple chatbot answers. An agent performs actions: creates a Jira ticket, checks order status in CRM, books a calendar slot. For orchestration, we use LangGraph: state graph where each node is a model or tool call. Vercel AI SDK useChat + tools for Next.js allows adding integration in 10 lines of code. Main challenge — reliability: the model sometimes calls the wrong tool or passes malformed parameters. Protection — Zod schemas for each tool and structured outputs to guarantee JSON.

What does the work include?

Additionally: we hand over vectorizer source code, monitoring dashboards (Langfuse), admin panel access for knowledge base updates. Post-production support — 1 month free.

What are the timelines?

Pricing is calculated individually after project discovery. We'll evaluate your project in 1 day. Contact us — we'll show how to turn AI from a toy into a profit-driving tool.