RAG OpenSearch: why knowledge base search returns the wrong results
A knowledge base assistant answers confidently and to the point — but not to the right question. The user asks how to process a return for a defective item, and gets the warranty repair policy: the words matched, but the meaning of the query stayed out of frame.
Irrelevant assistant answers are more expensive than an obvious error — people start acting on a plausible instruction.
The cause is usually the same: the search layer is built as an add-on over a full-text index. Lexical ranking by term frequency knows nothing about synonyms, paraphrases, or internal jargon.
If the base says “defect on acceptance” and the user asks “arrived damaged,” knowledge base search does not find the right clause, even though it is right there. The generator is not to blame here: it retells what it was given.
The bottleneck in the search layer is visible from measurements: the share of queries where the correct fragment appears in the top 5 turns out to be half of what is expected.
The more documents in the base that are similar in wording, the more RAG output quality drops — a neighboring topic is found, and the answer sounds coherent but misses the point.
This is closed not by replacing the model, but by rebuilding search: vector representation of the query and documents, hybrid output where lexical and semantic signals add up, filtering out irrelevant fragments with a reranking model, and document chunking with overlap. Let us break down how this works in OpenSearch and what it gives on a live knowledge base.
What a managed search layer over a knowledge base provides
A managed search layer is a set of settings, not “model magic.” We define which fields are indexed and with what weights, how the knowledge base is split, how the vector representation of fragments is built, and what exactly goes into the answer context.
That is why retrieval accuracy becomes a measurable quantity that can be managed, rather than luck based on how a question is phrased.
At the core is OpenSearch in hybrid search mode: the lexical part catches exact terms, part numbers, and abbreviations, while the vector part catches semantic paraphrases.
On top, candidate reranking and a threshold for cutting off weak matches work so that garbage does not enter the context. On a validation set of real questions, the share of relevant answers rises from 55–60% to 85–90%.
We separately address robustness to wording: synonyms, typos, different word order, and “in your own words” questions no longer throw search off course.
This removes an entire class of failures where the system confidently answers from the wrong knowledge base fragment — an error made more dangerous by the fact that it looks plausible on the surface.
We keep search response time within fractions of a second at the candidate selection stage; the main latency comes from answer generation, which we limit separately.
We take measurements on a fixed set of questions with reference answers, so after every settings change it is clear what exactly changed — in accuracy, speed, and the share of covered scenarios.
Search modes in RAG OpenSearch: what to choose
The retrieval mode determines whether the right fragment will be found when the question wording does not match the document text. Lexical search over an inverted index reliably catches part numbers, abbreviations, and exact wording, but loses on paraphrases.
A vector index finds matches by meaning, but gets confused by numbers and codes. In practice, the choice comes down to four options.
| Mode | When it fits | Limitations |
|---|---|---|
| Lexical search | Part numbers, codes, abbreviations, exact terms in the query | Does not find synonyms and paraphrases; recall drops on conversational questions |
| Vector index | Questions in one’s own words, conversational wording, multilingual corpus | Poorly distinguishes numbers and rare designations; reindexing when the embedding model changes |
| Hybrid search | Mixed queries where there is both meaning and an exact term | Requires a tuned weight for fusion: if skewed, one of the strengths is lost |
| Hybrid and results reranking | High requirements for the accuracy of the first three to five fragments; corpus is limited | Adds time to every query; requires a separate model and resources for it |
All four modes work on the same document collection: in OpenSearch, lexical and vector search are assembled over a single index, so switching is a matter of query configuration, not rebuilding storage.
Reindexing is needed only when the embedding model changes — we plan the migration in advance. We do not enable hybrid by default: we measure on real questions which mode puts the correct fragment in the top 5, and only after that fix the branch weight and the need for reranking.
Implementation stages: from corpus audit to handover
Implementing document search starts not with engine configuration, but with corpus analysis: until it is clear what is in the sources and how versions relate, any index configuration is guesswork.
Below is the order of work; some stages run in parallel if this does not hurt the verifiability of the result.
- Corpus audit and labeling. We consolidate sources, review formats, duplicates, and versioning. We label fragments — text, tables, attachments — and filter fields: document type, department, revision date.
- Control question set. Together with your team, we collect real questions with reference answers and sources. We fix the set: it serves as the quality benchmark until the end of the work.
- Index schema. We decide what goes into the index and in what chunks, which fields work in vector and text search, and how hybrid ranking is composed.
- Search configuration. We select the embedding model for the corpus language, fragment size, overlap, and hybrid weights — first on a sample, then on the entire array.
- Validation on control questions. We measure how often the correct source appears in the top positions and break down misses by cause: chunking, outdated revision, question outside the corpus.
- Adjustment and handover. We correct chunking, metadata, and filters until metrics stabilize, then hand over index configurations, the update policy, measurement scripts, and documentation to your engineers.
How each stage is completed
A stage is considered complete when there is a verifiable artifact: labeling, a question set, an index schema, a measurement report. That is why the system’s further operation does not depend on who exactly on our side led it.
Deliverables: indexes, schemas, update policies
The system remains operational without us: everything configured during RAG development on OpenSearch is handed over as reproducible artifacts.
The customer’s engineers can spin up a copy of the search, connect a new source, or change access rules without reverse-engineering someone else’s configuration from logs. The delivery includes the following.
- Index description: mappings of text and vector fields, representation dimensionality, analyzers, number of segments and replicas — with an explanation of why these values were chosen.
- Rules for splitting documents into fragments: window size, overlap, handling of tables and headings, so that a fragment preserves meaning and a reference to the source document.
- Hybrid search schema: contribution of lexical and vector components, order of merging and refinement ranking of results.
- Validation query set: reference questions with expected documents, so that after any change it is visible whether retrieval has degraded.
- Document access settings: how confidentiality labels and user groups enter the index and how the filter is applied at search time without letting through extra material.
- Knowledge base update policy: incremental reindexing of changed files, full rebuild schedule, handling deletions and old revisions.
- Support instructions: reading search logs, actions when answer quality drops, rollback to the previous index schema.
This is enough for the customer’s team to develop the solution independently, while we join in on a targeted basis — for expansion or complex changes.
Case study: support knowledge base, metrics before and after
The support knowledge base ran on lexical search: the query was split into words, and matches were sought by exact forms. Real-world phrasing broke it — “email doesn’t arrive” and “no email notifications” went to different results, and synonyms and typos were not handled at all.
Metrics were taken on the same sample: 200 real requests with reference articles. Validation on control questions gave the baseline: the needed document in the first three results — 41%, escalation to the second line — 38%, average time to answer — about 9 minutes.
What changed
Search was moved to OpenSearch in hybrid mode: lexical match plus vector representations of fragments, with the results of the two branches merged and reranked. Articles were chunked with overlap and assigned metadata — product, language, article type, freshness status.
The product filter triggers before ranking so that answers from another product line do not surface.
What the results were
The same control question set was rerun: top-three hit rate — 87%, escalations — 14%, time to answer — 2.5 minutes.
Rejection of false matches was added: if there is no suitable article in the base, the system returns an empty result with a note, rather than the first available instruction.
Error analysis after the run showed where base coverage was incomplete: some questions went into the queue for article writing, not into search refinement.
How hybrid search differs from pure vector search
Pure vector search converts the query into a numeric representation and retrieves the nearest fragments by meaning.
It handles paraphrases and questions in one’s own words confidently, but loses lexical items: a part number, part code, or error code dissolves in the averaged vector, and the output ends up with documents on the same topic but not the right ones.
Lexical search works inversely: it compares terms and finds exact matches, but fails when the question wording does not match the wording in the document.
Hybrid mode runs both queries over one index and merges the outputs — both semantic closeness and exact designation hits remain.
The scores of the two modes do not add directly: their scales are different.
Positions from both lists are recalculated into a unified rating, mode weights are set for the corpus, and for fields with codes and model names, boosting coefficients are added so that an exact match rises above semantic proximity.
On short one- or two-word queries, the vector is noisy; on highly specialized queries with abbreviations and hyphens, lexical search pulls through; on long natural formulations, the vector is stronger. Hybrid does not pick a winner in advance — it keeps both signals.
A separate task is query string parsing: a code must not be split on a hyphen and lose case, otherwise exact search stops finding the document.
Common technical doubts before starting
Engineering doubts before starting are rarely about “will search work at all” — they are about operations: how long to wait for an answer, what to do when the corpus updates, how not to return five copies of one paragraph, and whether the model can be changed later. Below are answers at the implementation level.
How much will hybrid search increase response latency?
Hybrid is a vector and lexical pass over the index with result merging, so search itself takes a small share of total response time.
The main time goes to generation, and it depends not on corpus size but on context volume: we fix how many fragments we send to the model, so knowledge base growth does not lengthen the answer.
We measure latency by stages — search, reranking, generation — and optimize the segment that actually lags.
How do you cut off duplicate documents and repeated fragments?
The cause is usually corpus preparation: the same text sits in several files or fragments are chunked with large overlap. On load, we compute a content fingerprint and discard duplicates, normalize formats, and set overlap so that neighboring parts do not compete with each other. One best fragment per source goes into the model context, not all similar ones.
Is full reindexing needed on every corpus update?
No. Each fragment is stored with a source identifier, so when a document is edited only its parts are reindexed, and deleted content is closed with a mark that excludes the fragment from results.
A full rebuild is needed if the vectorization model or chunking logic changes: we perform it in the background, in a separate index version, and switch traffic when the new version has been validated on the control set. Search remains available during this time.
Can the model be replaced without reworking search?
The index stores vector representations of text, and search returns ready fragments to any model through a unified interface — the layers are separated. Changing the generative model does not require a rebuild: it is enough to run the control question set and compare answers.
But changing the vectorization model means a different feature space — old vectors are incompatible with new ones, so a rebuild is inevitable here, and it is planned in advance as a separate stage.
Let’s discuss the task: what to send for assessment
Assessment of a search solution rests on two things — a real corpus and real queries.
We will collect part of the data at the first stage together with your team, but the more source data there is, the fewer assumptions in the RAG architecture on OpenSearch and the more accurate the timelines.
- Document corpus or export: formats (PDF, DOCX, tables, knowledge base), volume, languages, share of scans — this determines chunking and recognition.
- 50–100 real user queries in natural wording and expected answers for at least some of them — we measure quality on them before work starts.
- Relevance labeling, if it exists: which documents count as correct hits. Without it, we will collect metrics on an agreed sample.
- Access rules: who is allowed to see which documents, whether there is department separation. We embed permissions into the query itself, otherwise there will be leakage in the results.
- How search is structured now and what users complain about: queries without an answer, irrelevant results, lost documents.
- Environment and constraints: where we deploy the service, whether data transfer to external models is allowed, whether operation within the perimeter is required.
- Load and updates: how many queries per day, peak periods, acceptable response time, how often the corpus changes.
Based on this set, we will return with an assessment and work plan: indexing and chunking schema, hybrid logic — lexical search plus vector — reranking, stages, and acceptance procedure. Describe the task in free form; we will ask clarifying questions right away.
