RAG on ChromaDB: from prototype to production

Why a RAG prototype does not reach production operation

A RAG prototype is assembled on a folder of a dozen documents, and on that folder document search looks convincing: question — answer, with a source link next to it. The demo passes, and the solution seems ready.

Then the real corpus begins — thousands of pages, scans, nested tables, and three versions of the same policy at once.

The first fracture is data preparation. Fixed-size chunking splits a table or a contract clause in the middle, and half of a condition goes into the index.

The answer is formally correct in wording and incorrect in substance, and answer quality collapses not gradually but abruptly at a certain corpus volume.

The second is search itself. In the prototype it is the same for everyone, with no access separation: an employee receives a chunk from a document they should not see.

In production operation, filters are added to every query — permissions, department, version freshness — and after them relevance must be validated again.

The third is the update policy. Documents are alive: some change daily, some are revoked. The prototype is reindexed manually and in full, so the index quietly diverges from reality, and there is no way to notice it — there are no quality or coverage metrics.

A prototype validates an idea, but it does not withstand volume, access rights, and time. We examine how the transition to production operation is closed — from document preparation to quality control on a live stream.

What a managed RAG pipeline delivers in operation

A managed pipeline differs from a prototype in that every processing step is fixed in configuration and reproducible: document chunking, hybrid search, reranking of results, answer assembly, and mandatory source citation.

As long as these steps live in code, answer quality depends on who last edited the knowledge base, and it cannot be predicted.

Hybrid search instead of vector-only

Vector search handles semantic phrasing well, but it fails on exact SKUs, abbreviations, and internal policy numbers — and a significant share of queries arrives with them.

Hybrid search combines vector results with lexical results and passes the result through reranking: on a set of validation questions, the share of correct answers rises from about two-thirds to nine out of ten. Vague answers that the prototype hid behind a confident tone become visible and debuggable.

What is visible in operation

Response time to a question stays at one to two seconds because search runs over a prebuilt index, not over the entire document corpus. The share of escalations to a human specialist drops severalfold — mostly requests that simply are not in the knowledge base remain.

Knowledge base updates become a defined process: a changed document is reindexed in minutes, and the old version is not mixed into new answers.

The technical foundation for this is provided by ChromaDB: chunks are stored together with metadata — access rights, document version, and modification date — so filtering happens before search, not after. Next we examine what exactly we verify before moving the pipeline to production.

Search modes: vector, hybrid, with reranking

The search mode is chosen not by database size, but by how questions are phrased and how documents are structured.

Semantic queries — "how is a return processed," "what to do on a synchronization failure" — are confidently handled by vector search over embeddings.

As soon as SKUs, clause numbers, and abbreviations appear in the text, semantic proximity misses: the model does not distinguish "AB-1200" from "AB-2100."

In practice, modes do not replace but complement each other: hybrid retrieval raises several dozen candidates, and reranking leaves 3–5 truly relevant chunks that go into the model context.

In RAG on ChromaDB, this is configured at query level, so changing the mode does not require rebuilding the index — only the retrieval layer changes, not the data itself.

How we implement RAG: stages of engineering work

Before writing code, we analyze the data: what sources exist, in what format, how often they change, and who is responsible for accuracy. At this same step we assemble control questions — several dozen real employee queries together with the answers we consider correct.

Without such a set, there is nothing to discuss search quality with, and acceptance turns into an argument about tastes. Therefore each stage ends with an artifact that can be verified, not a report about work done.

1. Source inventory. We note the owner, format, and update frequency of each document — this determines the index recalculation schedule.
2. Pipeline schema. Before code, we draw the data path: how a document enters the index, how it is chunked, how it is updated, and how an outdated version is removed.
3. Search setup. We select the embedding model, chunk size, and hybrid retrieval — vector together with keyword search. We filter out irrelevant chunks, otherwise the model answers from noise.
4. Generation and citation. We assemble context and introduce a rule: the answer cites the source document, and at low confidence the system honestly says there is no data.
5. Validation on control questions. We calculate the share of correct answers and the share of fabrications; chunking and prompt wording are changed until the metrics reach the target.
6. Solution acceptance and handover to operation. We fix the metrics, give the team the pipeline schema, ChromaDB vector store settings, and the update policy. The on-call engineer can view logs and retrieval quality.

Delivery package: configurations, pipeline schemas, policies

We deliver the solution not as a service that cannot be touched without us, but as a set of artifacts: the customer's engineers deploy it in their own environment and change the generation model or document chunking rule independently.

Everything that determines system behavior in production operation lies in the repository and documentation — including what is usually lost during handover: how the index is updated and who has access to documents.

• Configurations — connection parameters to the vector store and model, chunk sizes and overlap, number of returned candidates: environment files, not values in code.
• Pipeline schema — the document path from upload and cleaning to chunking, vectorization, and writing to the collection, then search and answer generation; failure points are marked separately.
• Document access rules — role matrix, permission inheritance from the source, filtering at the search stage: the user does not receive a chunk they are not allowed to see.
• Index update policy — schedule for full and incremental reindexing, handling of modified and deleted documents, reconciliation of discrepancies with the source.
• Solution documentation — architecture description, local launch, list of answer quality and latency metrics, and actions for degradation.
• Set of validation questions with reference sources — for acceptance and regression after a model change, plus a rollback procedure: a collection snapshot and configuration version tied to the release.

Such a package removes dependence on the contractor: the system can be developed, validated, and handed over within the team without reconstructing logic from code.

Case study: knowledge base search for a support service

The support service of a service line maintained a knowledge base of 12,000 documents: policies, instructions on typical failures, and analyses of closed tickets.

The answer was searched manually — by file names and full-text search that understood neither synonyms nor customer wording.

On the first line, this gave 4–6 minutes to find an answer, about 40% of answers went out without source support, and disputed cases were escalated to the second line — a third of requests accumulated there.

What we did

We built a retrieval loop on ChromaDB: documents were chunked with 10–15% length overlap so the answer would not tear at a meaning boundary, and indexed.

Search runs in two steps — semantic similarity plus filters by document type, department, and freshness date, so an outdated policy does not override a current one.

A link to the original chunk is hardwired into the results: the operator opens the source and verifies the wording instead of trusting the model blindly.

We separately covered rollback: index versions are built as artifacts, and switching to the previous one takes minutes. If retrieval quality drops after a corpus update, we return without fully reindexing the database.

What we achieved

• Time to find an answer: 4–6 minutes → 25–40 seconds.
• Share of answers with source support: 40% → 92%.
• Requests to the second line: minus 35%.
• Questions resolved on the first line without escalation: 78%.

Some requests still go to the second line — these are non-standard incidents where an engineer makes the decision. But the operator arrives there with found sources and a draft answer, and the second line spends time on solving, not on correspondence.

How RAG differs from model fine-tuning

The difference is where knowledge is stored. In model fine-tuning, new information is fixed in the weights — the model's behavior changes as a whole, rather than the relevant chunk being inserted into a specific answer.

RAG solves the same task differently: external documents are stored in a repository, and for each query the system retrieves relevant chunks and passes them to the model along with the question.

Hence the difference in knowledge updates. In a retrieval loop, it is enough to load changed documents and recalculate indices — an edit in a policy or instruction appears in answers the same day, without stopping the service.

Model fine-tuning is a separate run on a prepared set, a degradation check, and the risk of forgetting what was previously learned: the model starts answering better in the new domain and worse in the old one. Rollback is possible only to the previous checkpoint together with its entire worldview.

Fine-tuning is justified when what needs to change is not facts but behavior: maintaining answer style and structure, working confidently in narrow terminology, performing a same-type task — classification, field extraction, or routing requests by category.

RAG or fine-tuning is also chosen based on verifiability requirements: in a retrieval loop, you can see which chunk the answer was assembled from, and the source link can be checked without interpreting weights.

In practice, a combination more often works: the model is fine-tuned on format and domain, while knowledge is kept in ChromaDB and indexing is updated on every document edit.

Then the share of answers with a source link grows, and knowledge update time is measured in hours, not retraining cycles.

Technical concerns: quality, data, access

Before the start, the same five questions almost always come up. Below are short substantive answers: they determine whether the system will bring benefit or remain a demo.

How is the knowledge base updated?

Indexing is incremental: if a document changed, only it is recalculated. A deleted document leaves retrieval together with its chunks, so revoked instructions do not surface in answers.

Are access rights enforced?

The permission filter is applied before ranking: search sees only chunks available to the employee, so a restricted section cannot appear in an answer. Logs record which chunks were used.

What if we need to change storage?

The access layer is separated from storage: vectors and descriptions are exported, and the metadata schema is preserved. Migration becomes a planned operation, not a new project.

Can an unsuccessful update be rolled back?

Before reindexing, a snapshot is taken, and the pointer is switched by a separate command. If quality drops, rollback takes minutes, and search continues to work on the previous data.

Let's discuss your task: where we start

A project estimate is based on your data, not on our assumptions: corpus volume and access rights determine document chunking, the search method, and which quality checks make sense at all.

To discuss the task substantively and return with an estimate and work plan, we need this minimum input data from you:

• A corpus of documents in original formats — an export for a representative period, not one "showcase" folder: on five files you cannot test how search behaves on duplicates and contradictions between versions of a policy.
• Update frequency and data owner — quarterly manual updates or a continuous stream from an internal system: this determines whether we do a one-time load or regular reindexing with removal of outdated content.
• Access rights: who can see which sections and whether retrieval must be filtered by department or role. This affects the storage schema and where exactly the filter is applied.
• 15–30 real user questions with expected answers and source indication — on this sample we fix the acceptance criterion and measure quality, rather than arguing about it in a demo.
• Integration points and environment constraints: where search returns the answer (chat, internal portal, operator workstation), whether data may go to external services, or everything must work in a closed environment.
• Current difficulties in numbers: how much time answer search takes now, what share of requests remain unanswered, and which errors repeat most often.
• Readiness criterion: the share of questions where the answer is found and confirmed by a document link, and the acceptable response time.

Describe the task in free form and attach what you have at hand — we will clarify the rest in conversation.

We will return with an estimate and work plan: stages, content of each, responsibility area of your team, and points where you can stop and revise scope without losing what has already been done.