Implementing RAG for AI Chatbots
RAG solves a specific problem: the model doesn’t know your product, your documentation, your internal regulations. Fine-tuning is expensive and slow to update. RAG is cheaper, more current, and transparent. User asks a question → system retrieves relevant document fragments → passes them into the model’s context → model answers based on real data. Our experience shows that a well-implemented RAG boosts answer accuracy to 95% and reduces support load by 40–50%.
Unlike traditional chatbots, an RAG bot uses a live knowledge base: you add or change documents — the bot immediately answers using updated information. This is especially important for mobile apps with frequently changing tariffs, products, or terms. We have deployed RAG systems for 15+ projects, including mobile apps with sensitive data. Our team has 8+ years of experience in mobile development (iOS, Android, Flutter) and server architecture. We guarantee transparency: you see where each answer comes from. Average RAG bot response time is 1–3 seconds, including retrieval and generation, meeting user expectations.
The technique is described in Lewis et al.
Problems RAG Solves
The first and foremost — model hallucinations. Without context, an LLM can invent nonexistent features or parameters. RAG anchors answers to your documentation.
The second — data freshness. Retraining a model every time documentation changes is expensive and slow. RAG pulls the latest version from the knowledge base.
The third — user trust. A mobile chatbot with RAG shows answer sources — users can verify the information. This reduces escalations.
Components of an RAG System
Ingestion, Retrieval, Generation — how they connect
Ingestion (loading and indexing):
- Document chunking
- Creating embeddings for each chunk
- Storing in a vector database
Retrieval:
- Embedding the user query
- Vector search (cosine similarity / ANN)
- Optional reranking
Generation:
- Building a prompt with context
- Calling the LLM
- Postprocessing the answer
On mobile, all ingestion and most retrieval are server‑side. The client makes an API request and receives an answer with sources.
Chunking: The Most Underestimated Step
RAG quality depends on chunk quality. Poor chunking kills accuracy regardless of the model.
Fixed chunking (500 characters each) — don’t do this. It breaks sentences and loses paragraph context.
Semantic chunking — splitting by meaning boundaries (headers, paragraphs, sentences). For Markdown and HTML it works out of the box. The LangChain4j library for Java/Kotlin provides RecursiveCharacterTextSplitter with delimiters ["\n\n", "\n", ". "] — that’s the right approach.
Overlap — 10–20% overlap between chunks: include the last 50–100 tokens of the previous chunk at the start of the next. This preserves context at boundaries.
Optimal chunk size depends on document type: for technical documentation — 300–500 tokens, for legal texts — 500–800 tokens, for FAQ — one question+answer per chunk.
Which Embedding Model to Choose?
| Model | Dimensions | Context | Cost | Best for |
|---|---|---|---|---|
| text-embedding-3-small | 1536 | 8192 | Cheap | General content |
| text-embedding-3-large | 3072 | 8192 | Medium | Technical documentation |
| nomic-embed-text | 768 | 8192 | Free (self‑host) | Private data |
| multilingual-e5-large | 1024 | 512 | Free (self‑host) | Multilingual content |
For a mobile app with sensitive data — use a self‑hosted model. OpenAI Embeddings send your documents to OpenAI servers.
Hybrid Search or Pure Vector?
Hybrid search — combining vector search with BM25 (keyword search) yields better results than vector search alone. pgvector + pg_trgm let you do this in PostgreSQL without extra infrastructure.
Reranking — after vector search, take the top 20 results, run them through a cross‑encoder model (cross-encoder/ms-marco-MiniLM-L-6-v2), return the top 5. This greatly improves relevance. Cohere Rerank API is an alternative if you don’t want to self‑host.
Metadata filtering — if documents have metadata (date, section, language, type), filter by them before vector search. Searching vectors among 10k relevant chunks instead of a million is faster and more accurate.
Building the Context Prompt
System: You are a company product assistant. Answer ONLY based on the provided context. If the answer is not in the context, say so explicitly.
Context:
[Chunk 1]: <text>
[Chunk 2]: <text>
[Chunk 3]: <text>
User: How to set up two‑factor authentication?
Including sources is good practice. On mobile we display a list of chunks/documents below the answer: users can verify where the information comes from. This reduces hallucination risk and builds trust.
Mobile UI for an RAG Bot
Answer rendering specifics:
- Streaming via SSE — answer appears gradually
- Sources below the answer (collapsible list)
- “Searching knowledge base” indicator during Retrieval (100–300 ms)
- “Answer not found” button for escalation to operator
On Flutter: flutter_markdown for rendering, custom widget for sources. On iOS: UILabel with NSAttributedString or UITextView + WKWebView for Markdown. On Android: Markwon — the best Markdown renderer for RecyclerView.
Scope of Work for RAG Implementation
- Knowledge base corpus audit and indexing scheme design
- Vector DB selection and deployment (pgvector, Qdrant, Pinecone — based on your stack)
- Ingestion pipeline with semantic chunking and overlap
- Hybrid search + reranking setup for maximum relevance
- LLM integration (OpenAI, GPT‑4, Claude, YandexGPT, or self‑hosted)
- Mobile chat UI with streaming, sources, and escalation
- Operations documentation and team training
- Quality evaluation using RAGAS metrics
Order a knowledge base audit — we will analyze your documents and propose an RAG implementation plan. Contact us for a consultation. Write to us for a detailed plan.
Timelines and Estimates
| Phase | Duration |
|---|---|
| Audit and design | 1 week |
| Pipeline implementation | 2–3 weeks |
| Integration and UI | 2–4 weeks |
| Testing and refinement | 1–2 weeks |
A basic RAG bot with simple documentation — from 3 weeks. A production system with hybrid search, reranking, multilingual support, and quality evaluation — up to 12 weeks. Exact timelines depend on document volume and integration complexity.
Common Mistakes When Implementing RAG
- Ignoring document preprocessing: PDFs with images, scans, tables — require OCR and cleaning.
- No metadata: without filtering by date or section, search quality drops.
- Insufficient testing: use RAGAS metrics — precision, relevance, faithfulness.
- Neglecting security: configure context filtering so the model does not expose sensitive data.
RAG is not a single model but a system of many components. Contact experienced specialists: we can help avoid common pitfalls and deploy a solution that works reliably.