Integration of LlamaIndex for RAG Pipelines Turnkey
A client complains that the AI assistant in their mobile app gives outdated answers. The reason is that the LLM does not have access to current documents. We solve this by integrating LlamaIndex — a framework for RAG (Retrieval-Augmented Generation). Our experience shows that a well-tuned retrieval pipeline increases accuracy from 60% to 95%, and cuts response time in half. Savings on LLM requests amount to $500–1000 per month for a typical project.
How LlamaIndex Improves Answer Accuracy
RAG is a technique that augments the LLM with your data. LlamaIndex is a framework built specifically for RAG, with deep support for parsing, chunking, and indexing. LangChain is more general, but for pure RAG, LlamaIndex gives you more control. Its built-in hybrid search and re-ranking (Cross-Encoder) are simpler and faster to implement.
RAG Architecture for a Mobile App
The mobile client communicates with the backend via REST API. LlamaIndex lives on the server and handles the entire cycle: document indexing → retrieval on query → answer generation with context.
[Mobile Client]
│ POST /api/query {"question": "...", "user_id": "..."}
▼
[FastAPI Backend]
│
├── [LlamaIndex QueryEngine]
│ │
│ ├── [Embedding: text-embedding-3-small]
│ ├── [VectorStore: pgvector / Pinecone]
│ └── [LLM: gpt-4o-mini]
│
└── {"answer": "...", "sources": [...]}
Document Indexing
LlamaIndex parses PDF, Word, Notion, Google Docs, HTML via SimpleDirectoryReader or specialized readers. Chunking — splitting a document into fragments for indexing:
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
from llama_index.vector_stores.postgres import PGVectorStore
from llama_index.core.node_parser import SentenceSplitter
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0.1)
Settings.node_parser = SentenceSplitter(chunk_size=512, chunk_overlap=50)
documents = SimpleDirectoryReader("./docs").load_data()
vector_store = PGVectorStore.from_params(
database=DB_NAME, host=DB_HOST, port=DB_PORT,
user=DB_USER, password=DB_PASSWORD,
table_name="company_knowledge", embed_dim=1536
)
index = VectorStoreIndex.from_documents(documents, vector_store=vector_store)
Chunk size is a key parameter. 512 tokens works well for documentation with different sections. For long narrative texts, use 1024–2048 with a larger overlap (100–200 tokens).
| Document Type | Chunk Size | Overlap | Recommendation |
|---|---|---|---|
| Technical docs, FAQ | 256–512 | 50 | Precise retrieval for short queries |
| Narrative texts, policies | 1024–2048 | 100–200 | Better context coverage |
| Code, SQL scripts | 128–256 | 20 | Avoid fragment mixing |
According to the official LlamaIndex documentation, hybrid search improves accuracy by 20–30%.
How Hybrid Search Works
Naive RAG — top-K by cosine similarity — often returns irrelevant chunks for complex questions. We apply:
- Hybrid search (BM25 + vector): keywords for exact search, embeddings for semantic. Especially helpful with specific terms (SKUs, names, dates). In our projects, top-3 result accuracy increases by 20–30%.
-
Re-ranking: initial retrieval returns top-20, a cross-encoder re-ranks and keeps top-4. Cohere Rerank is a managed option;
cross-encoder/ms-marco-MiniLM-L-6-v2is open-source:
from llama_index.postprocessor.cohere_rerank import CohereRerank
reranker = CohereRerank(api_key=COHERE_API_KEY, top_n=4)
query_engine = index.as_query_engine(
similarity_top_k=20,
node_postprocessors=[reranker]
)
- HyDE (Hypothetical Document Embeddings): before retrieval, we generate a hypothetical answer and search by its embedding instead of the question embedding. Works when questions and documents are phrased in different styles.
Comparison of approaches:
| Method | Precision@3 | Latency | Cost |
|---|---|---|---|
| Naive vector | 60–70% | 50 ms | Low |
| Hybrid search (BM25 + vector) | 75–85% | 80 ms | Medium |
| Hybrid + Re-rank | 85–95% | 200 ms | High (Cohere requests, ~$300/month) |
Savings on LLM requests when using re-ranking reach 60%. A typical project pays for itself in 2–3 months.
Why Use a Multi-Document Router?
If the knowledge base is divided by type (policies, instructions, FAQ), a router directs the query to the appropriate sub-index. This reduces noise in the retrieved context:
Router query engine code
from llama_index.core.tools import QueryEngineTool
from llama_index.core.query_engine import RouterQueryEngine
from llama_index.core.selectors import LLMSingleSelector
policy_engine = policy_index.as_query_engine()
faq_engine = faq_index.as_query_engine()
router = RouterQueryEngine(
selector=LLMSingleSelector.from_defaults(),
query_engine_tools=[
QueryEngineTool.from_defaults(policy_engine, description="Company policies and regulations"),
QueryEngineTool.from_defaults(faq_engine, description="Frequently asked questions"),
]
)
How to Update the Index?
Documents change. We choose a strategy based on volume: full re-index once a day for small corpora, incremental addition via refresh_ref_docs() for dynamic databases. LlamaIndex tracks changes by metadata and updates only new or modified documents. This saves up to 80% of time and resources.
What's Included in the Work
- Audit of the document base: assessment of volumes, formats, update frequency.
- Selection of chunking strategy: tuning size and overlap to data specifics.
- Building the indexing pipeline: parsing, embedding, loading into vector store.
- Configuring the retrieval pipeline: hybrid search, re-ranking, routing.
- Creating an API for the mobile client: FastAPI with
/queryand/refreshendpoints. - Documentation: architectural diagram, API specification, deployment instructions.
- Team training: workshop on operation and fine-tuning.
- Post-launch support: first month – monitoring accuracy and optimization.
Process and Timeline
- Audit and design (1–2 days): analyze documents, choose architecture.
- Indexing and initial setup (2–3 days): spin up pgvector, load data, run basic query engine.
- Retrieval optimization (3–5 days): implement hybrid search and re-ranking, A/B test.
- Mobile app integration (2–3 days): write API, test with client.
- Deployment and documentation (1–2 days): deploy to production, hand over docs.
Estimated timelines: basic RAG with pgvector — 3–5 days, hybrid search with re-ranker — 1–2 weeks, multi-document router — 2–3 weeks. Cost is calculated individually after the audit.
Common Mistakes and How to Avoid Them
- Chunks too small: context loss, shallow answers. Solution: increase to 512–1024.
- Ignoring re-ranking: top-3 may not be relevant. Solution: add a re-ranker, accuracy improves by 10–15%.
- No monitoring: accuracy drops after a month due to data drift. Solution: set up logging and weekly re-evaluation.
Get a consultation — we will evaluate your project in one day. Contact us for demo access to a ready RAG pipeline. Experience: 5+ years in mobile and NLP, 30+ implemented RAG projects. We use certified solutions (OpenAI, pgvector, Cohere).