RAG development with Milvus vector database — experience and cases

RAG Milvus: why document search returns irrelevant answers

Irrelevant answers from the assistant are almost never related to the quality of the model itself.

The query “what are the warranty terms for delivery to a branch” goes to keyword search, while the regulation states “the warranty period is calculated from the date of acceptance”: the meaning matches, the wording does not.

No match is found, the assistant receives empty context or a random fragment and fills in the answer with a guess. Externally this looks like a model hallucination, although the cause lies in the search layer over internal documentation.

The second cause is fragmented sources. Regulations live in a file archive, terms and details — in an accounting system, and part of the knowledge settles in correspondence and notes.

Until documents are brought to a unified form while preserving headings, numbering, and links between sections, search sees flat text and loses context. The same term in two departments means different things — keyword matching does not distinguish this.

The third and most common one is the absence of vector representations of documents. Without chunking with overlap, without vectors and an index over them, knowledge base search relies on exact word matches rather than meaning.

A rephrased question, synonym, abbreviation, or internal acronym — and the needed document is not found, even though it physically resides in the storage. Answer quality on domain data degrades most where the terminology is in-house and does not match generally accepted terms.

This is where Milvus comes in: the database stores representations of fragments and finds the nearest by meaning, not by letters, with filtering by attributes — department, document type, revision date.

But the database itself does not fix source markup, does not decide how to cut documents, and does not check whether the truly needed fragment made it into the context.

Next we break down how this is addressed: from the chunking scheme and indexing strategy to relevance evaluation on real domain questions and rollback to the previous index version.

What changes after implementation: accuracy, speed, search resilience

Before implementation, assistant quality was measured “by eye”: answers looked plausible, but on some questions the system confidently retold an outdated document.

We established a baseline on a set of real inquiries — the share of answers where the found fragment actually contained the answer was about two thirds.

After configuring document chunking, hybrid search, and candidate reranking, it rose to values where manual review is needed only in a small number of complex cases.

The key here is not a one-off measurement, but a regression set of questions that is run on every configuration change: without it, any improvement quietly rolls back within a month.

Response time as the database grows behaves nonlinearly if the index is not managed. On Milvus we separate collections by purpose, keep search parameters per segment, and rebuild the index on a schedule rather than at release time.

This provides steady response on an order-of-magnitude larger volume: search remains within one to two seconds, and growth in the number of documents no longer turns into growth in latency.

Resilience to rephrasing is a separate metric that customers usually do not check before launch.

The same question asked in three ways previously led to different fragments; now the output matches by meaning due to a combination of vector and full-text search with metadata filters.

We measure this by the share of matching top results across paraphrases — it stays above ninety percent.

Output quality control becomes a regular procedure: once per sprint we run the set, look at failures, add them to tests, and only then change the logic. This way assistant answer accuracy stops being a matter of luck and becomes a parameter that can be maintained.

Options for building knowledge base search: comparison of approaches

The choice of search scheme is determined not by fashion, but by the type of queries coming into the knowledge base. Lexical keyword matching finds exact matches — SKUs, codes, regulation names — but fails where the user phrases the question in their own words. Vector search catches meaning, but may miss a rare exact token on which the answer depends.

Between these extremes stands hybrid search: lexical and vector ranking are computed in parallel and merged by weights. Below are three schemes that are realistically chosen between at project start.

Indexing the knowledge base in any option begins with cutting documents into fragments and metadata: source, version, access rights. We test the scheme on real queries — what share of needed documents makes it into the top 5 and how often there is no correct answer at all.

An external vector database, such as Milvus, handles storage and vector search, but the choice of ranking mode and the quality of fragmentation remain with the team.

How we conduct implementation: stages of engineering work

Implementation begins not with code, but with an inventory of knowledge.

Answer quality is determined by what underlies search and how the corpus is split into fragments, so the order of work is fixed: first source audit, then indexing scheme, then search configuration, and only after that acceptance. Each stage leaves an artifact by which the solution can be verified and handed over for support.

1. Data source audit. We collect a list of systems, file formats, access rights, duplicates, and versions. Separately, we record how often documents change and what to do with outdated revisions — otherwise search will return canceled orders and old regulations.
2. Document indexing scheme. We define rules for splitting into fragments and overlap between them, the composition of metadata (department, type, date, access level), and a vectorization model for the corpus language. At this step we establish role-based filtering: an employee receives only the documents to which they have access.
3. Building the vector store on Milvus. We mark up collections, enable versioning, and a separate loop for loading updates so reindexing does not stop search. This way a regulation changed in the morning appears in output without service downtime.
4. Search configuration. We assemble hybrid output: vector proximity plus metadata filters and reranking. We set a cutoff threshold — if there are no suitable fragments, the system honestly reports this instead of answering by guess.
5. Acceptance on control questions. We prepare a set of reference pairs “question — correct source” together with your experts and measure source hit and answer completeness. We analyze misses: some are closed by changing chunking, others by refining metadata.
6. Handover and support. We provide the index update regulation, quality dashboard, and procedure for when a new knowledge source appears.

The implementation order is structured so that each subsequent stage relies on the verified previous one — therefore contentious points are visible before launch, not after.

What we hand over to the customer: configurations, schemes, and regulations

Putting the system into operation means handing over not only code, but everything needed for independent operation. Below is a list of artifacts that the customer receives as output.

Each item is checked together with the customer’s team at acceptance: we show where it is located, how it is applied, and what will happen if it is changed.

• Working configurations of the search loop — document chunking, vector representation parameters, selection rules, and relevance thresholds. Each value is accompanied by a measurement: how it affects output completeness and accuracy.
• Data flow diagrams — the path of a document from source to answer: extraction, cleaning, vector construction, writing to Milvus, querying the model. The diagram marks queues, retries on failures, and points where data is filtered by rights.
• Document access policies — separation at the collection and field level, inheritance of rights from the corporate directory, anonymization rules. The answer is assembled only from fragments available to a specific user.
• Index update regulation — when additional training on new documents is launched, when a full rebuild happens, how integrity is checked, and how to roll back to the previous state.
• Support instructions — what to look at in monitoring, which metrics to consider alarming, what to do when answer quality degrades, whom to call.
• A set of verification questions with reference answers — run after each index rebuild, shows whether output has degraded on edge cases.
• Dependency and access map — external services, accounts, certificates, their validity periods, and owners on the customer side.

Acceptance follows a scenario: the customer’s engineer independently adds a document, waits for the index update, and asks a question — without our help. If the step does not pass, it means the documentation handover is incomplete, and we refine it to that state.

Case study: how we reduced the share of irrelevant answers threefold

The customer is a distributed knowledge base: regulations, contracts, and technical documentation from several dozen departments, around 12 million fragments in one index.

The assistant searched by vector proximity with the same top for any question — without accounting for department, document type, or validity period.

At the start, the share of irrelevant answers held at 21%, and in a third of cases the link led to a canceled revision: drafts and approved versions lay in the same collection, and fragments from neighboring domains overlapped in meaning. Users stopped opening links and returned to manual folder search.

What we rebuilt

We separated the index into subject collections with common metadata fields: department, document type, validity period, access level. Filters on them are applied before search, rather than cutting off what has already been found.

We cut documents by structure — heading, clause, table — while preserving the parent section so the fragment does not lose context.

We rebuilt ranking: to vector search in Milvus we added lexical search, merged the outputs, and reranked the top 50 fragments with a separate model trained on support service labels — which sources employees actually opened.

Metrics before and after

Project result: irrelevant answers dropped threefold, response speed almost did not change — reranking the top fragments adds fractions of a second and fits within the dialogue budget.

Index updates were switched to incremental assembly: editing one document recalculates only its fragments; a full rebuild is needed only when the chunking strategy changes.

How does RAG Milvus differ from ordinary keyword search?

Classical search works on an inverted index: a document appears in the output if it contains query lexemes, taking word forms into account. Selection is by exact match, so the result is predictable and response is minimal.

But the query “how to get a refund for a canceled trip” will not find the article “refund upon booking cancellation” — the words do not overlap, although the meaning is the same.

Vector search converts the query and documents into embeddings — numeric representations where distance reflects semantic proximity. Milvus stores such vectors and, by cosine similarity, selects the nearest fragments rather than strings with the needed words. The wording may not match the document text at all — the topic matters, not literal lexeme overlap.

Next, fragment selection comes into play. The knowledge base is cut into semantic chunks of several hundred tokens with a small overlap so the thought is not broken at the boundary.

The nearest fragments are taken from the output, if needed with a metadata filter: section, date, access rights. The result is not an entire page, but the specific paragraph where the answer lies.

The key difference shows in the result. Keyword search returns a list of links, and the person reads, compares, and synthesizes the conclusion.

The RAG loop inserts the found context directly into the request to the language model, and it forms a coherent answer based specifically on these fragments, not on general knowledge from training.

In practice, approaches are combined: the lexical ranker BM25 plus vector search over the database, outputs merged with weight recalculation. Exact terms, SKUs, and names are better caught by words; paraphrases and synonyms — by vectors.

The price of the hybrid is maintaining two indexes and recalculating them when the embedding model changes.

Technical concerns: limitations, risks, and quality control

RAG does not remove the data quality issue: if the corpus contains two revisions of a regulation, search will honestly return both, and the answer will be contradictory.

On the vector database Milvus, we address this with metadata — version, date, department, access level — and filtering at the search stage, not after generation.

Quality control rests on measurements, not promises. We record the share of answers with a source link and the share of correct refusals “no data,” and run the reference question set on every index update — degradation is visible before rollout.

How are access rights to sections separated?

The metadata filter triggers before generation — otherwise a restricted fragment would already have entered the model context. Rights are synchronized with the corporate directory, and changes are picked up without a rebuild.

How often is the index updated?

Incremental reindexing runs on document change events; a full rebuild runs on a schedule. Canceled versions are deleted so search does not return an outdated clause.

What about the source documents and what happens after handover?

Before indexing, we inventory the corpus and deduplicate; we record schema limitations at the start. We hand over collection schemas, access policies, and verification questions — the customer’s team changes the corpus itself.

Let’s discuss the task: scope of work and what we need from you

So that the assessment of the RAG solution and the work plan rely on facts rather than guesses, the first conversation needs a minimal set of input data.

The more specifically the document corpus and work scenarios are described, the more accurately we will state the scope of work, timeline, and starting order — without clarifying email loops.

• 15–20 real user questions as they are asked in work, with the expected answer and a link to the source document.
• a list of knowledge sources: file types, volume, languages, whether there are scans and tables, how often content is updated.
• how the base is replenished — manually, by export from an accounting system, or on a schedule: this determines the index update scheme and the place of Milvus in the loop.
• access requirements: test environment, read rights to sources, restrictions on personal data and closed sections.
• an example answer you consider correct: a quote from the document, a short summary, or step-by-step instructions.
• load profile: how many employees and requests per day, whether a real-time answer is needed.

With this set, we come back with an assessment and plan: breakdown into stages, what we do at each, where your decisions are needed, and when the first working prototype on your data will appear.

A substantive conversation usually takes less than an hour and ends with specifics on scope of work and timeline — describe the task, and we will come back with a plan.