Why the choice of embedding model for RAG determines answer quality
RAG works in two stages: first search over an internal knowledge base, then generation of an answer based on retrieved fragments.
Answer quality is limited by retrieval quality — if the first step returned wrong pieces, the model fills in text with general ideas about the world rather than your documents.
The difference is set by the embedding model: it decides what counts as semantically close. A weak model catches matches of rare words and abbreviations instead of meaning — irrelevant fragments, neighboring sections, or outdated versions end up in the top. This then shows in the answer: it sounds confident but relies not on the source but on a guess.
The cost of such an error accumulates unnoticed. Users stop trusting search over the knowledge base and return to manual file browsing, and the team edits wording after every output.
A separate matter is the frequency of corpus reindexing: how the model chunks text into fragments and calculates similarity determines how much data must be recalculated when documents update and when the model itself changes.
Therefore, the choice of embedding model for RAG is not an implementation detail but an engineering decision with measurable consequences: share of relevant fragments in the top, share of answers with a document reference, time of full reindexing after corpus edits.
How we select a model for a specific corpus and how we verify the result before migrating to it — we cover further.
What a correctly selected model delivers: measurable effects
We take the first metric — share of relevant fragments — on our own set of 100–200 real requests: whether the correct fragment is in the top five search results.
A model selected for the language and terminology of specific documentation raises this share from about half of queries to 85–95%.
We take general open rankings such as MTEB as a reference, but the measurement on the customer's data remains decisive — they do not know your abbreviations and internal vocabulary.
Robustness to paraphrasing is checked separately: we ask the same question in 6–8 formulations, including colloquial ones and with typos, and look at the variance of retrieval.
If the model is selected correctly, the set of fragments almost does not change — the answer does not fall apart from word order or a synonym, and the user does not need to guess the "correct" wording.
Answer preparation time is determined by search, not generation.
Vector dimensionality and index construction method determine whether fragment selection fits into fractions of a second as the database grows severalfold; we keep two routes — fast over short vectors and precise recalculation for complex queries — and reuse already computed vectors for repeated questions.
The reverse effect is visible in the volume of manual edits: when the right fragments enter the context, the model less often fills in facts, and the operator edits one answer out of ten instead of every second one.
We record "before" and "after" measurements in the report together with the methodology — they can be repeated on new data and used to verify that the gain is not one-time.
Comparison of approaches: ready-made embedding, fine-tuning, hybrid
The three paths differ not in "model quality" but in how much domain signal must be embedded in vectors.
A ready-made general-purpose model has already seen heterogeneous texts and works out of the box; fine-tuning on a domain corpus shifts vector space toward your terminology; a hybrid scheme adds lexical search to semantic search and covers exact matches.
| Option | When it fits | Limitations |
|---|---|---|
| Ready-made general-purpose model | Corpus up to tens of thousands of fragments, terminology close to general language, multilingual corpus needed | Does not distinguish internal synonyms and codes; quality hits the domain ceiling |
| Fine-tuning on a domain corpus | Stable terminology, many similar documents, there is data to label "query — fragment" pairs with | Requires a cleaned corpus and labeling; rollback to previous version is a separate procedure; a multilingual corpus easily loses its second language |
| Hybrid search over documents | Queries with part numbers, abbreviations, exact formulations | Two indexes and selection of fusion weight for retrieval — volume of checks and support time grow |
The corpus dictates the choice, not model size. If documents contain part numbers and internal codes, semantic search will not retain them — a hybrid is needed.
If terminology is stable and uniform, fine-tuning pays off; on a multilingual corpus it requires separate validation, otherwise the second language degrades.
Therefore, we start from measurement: we collect "query — relevant fragment" pairs and calculate the share found in top positions.
If the ready-made model maintains the bar — we do not start fine-tuning and do not spend the corpus on a task that chunking into fragments and reranking retrieval can solve.
Implementation stages: from corpus audit to handover to operations
The order of work is the same both for a new corpus and for replacing the embedding model in a running RAG search: first we measure, then we change.
The main risk is choosing a model based on someone else's comparison tables: on a demo set the difference between candidates is unnoticeable, but on scans, tables, and outdated versions it is exactly what determines whether an employee finds the answer or not.
- Corpus document audit. We count volume, formats, share of scans and tables, duplicates, outdated versions. Output — source map and rules for splitting text into fragments.
- Labeling validation questions for search evaluation. 50–100 real employee questions with a reference answer and a link to the source document. The set does not change thereafter — otherwise measurements are incomparable.
- Comparison of models on your corpus. We run candidates on one set and look at the share of hits of the required source in the top five, separately — quality on short and imprecise queries.
- Indexing setup. Fragment size and overlap, metadata (version, department, validity period), combination of vector and exact search, reindexing schedule when documents update.
- Calibration of the "no answer" threshold. We select at what similarity the system answers and at what similarity it honestly reports that there is no data. This removes confident fabrications on gaps in the corpus.
- Search quality acceptance. We fix target metrics and run the control set. A discrepancy is a reason to return to chunking or the model, not to rewrite questions.
- Handover to operations. Configuration, reindexing procedure, monitoring share of failures, access differentiation, and training of the customer's team.
Handover artifacts: indexing schemes, procedures, documentation
A solution with corpus search and answer generation lives longer than the implementation project itself: corpus grows, embedding model updates, access requirements change.
Everything not fixed in schemes and procedures remains with the contractor and becomes the only way to fix something. Therefore, we assemble the handover package as a working tool for the customer's team.
- Corpus indexing scheme: sources, rules for splitting into fragments, window size and overlap, metadata composition (source, date, access level) alongside the vector.
- Rationale for embedding model selection: which options were compared on the control question set and by which metrics — share of hits of the required source in top retrieval.
- Index update procedure: reindexing triggers, incremental update versus full rebuild, build time, actions on failure.
- Data access policies: transfer of rights from document to fragment, filters during search, what happens to vectors when access is revoked.
- Rollback plan: compatibility of index and model versions, return to previous build without losing fresh documents.
- Instructions for the internal team: where configurations are stored, how to add a source, how to start reindexing, which checks are mandatory before rollout.
- Request path scheme: question normalization, search, filtering, reranking, call to the language model — indicating the place of each step.
Acceptance follows the same list: the customer's engineer goes through the procedures on their own test bench, and we close discrepancies before handover. After that, the pipeline develops inside the company, without mandatory dependence on us.
Case study: moved knowledge base search to a new model
The customer is a service company with a knowledge base of several thousand documents: regulations, contracts, equipment instructions.
Search worked on vector representations trained on general texts: on everyday questions retrieval was decent, but on internal terminology top results included semantically close yet incorrect fragments.
Support employees manually checked the answer against the primary source — each answer took minutes.
We compared candidates on the public document search quality ranking MTEB, selected three models, and ran them on our own set of several hundred real questions with reference fragments.
We chose not by average score but by the share of hits of the required fragment in the top three — on our terminology the order of leaders turned out different.
What we changed in the search pipeline
We tied document chunking to structure — heading, clause, table — added overlap at boundaries and hybrid retrieval: vector plus keyword search, with reranking of top results. Unchanged files are no longer recalculated: updating a document affects only its parts.
| Metric | Before | After |
|---|---|---|
| Correct answers without manual verification | 61% | 92% |
| Knowledge base search time | 4–6 s | less than 1 s |
| Document update in the index | ~40 min | ~3 min |
A month after launch, the share of requests where an employee went to check the primary source fell from 40% to 8%. We hand over measurements and configuration together with the solution — the result can be repeated on your own knowledge base.
What to do if the vector index is already built on another solution
Reusing an index built on another model directly is impossible: vector dimensionality, similarity metric, and cutoff threshold are tied to a specific model, and old vectors lose meaning in the new space.
The question is not whether recalculation can be avoided, but how to perform corpus reindexing so that retrieval does not degrade and the service does not go down.
Changing the model without stopping the service rests on one technique: the model name and index version are stored in the collection metadata, and retrieval reads them from configuration, not from code.
Then switching is a value change, rollback is returning the previous one, and rebuilding requires no edits in search.
Technical concerns: corpus volume, languages, data updates
Five concerns repeat at every assessment — we resolve them before launch, not after the first user complaints. Briefly: what we check on a pilot document sample and what reserve we build into the index architecture.
Will the index be enough for the entire corpus and how to chunk documents?
The limiter is not volume but fragment size: large blurs meaning, small loses context. We keep 1-2 paragraphs with overlap and check the reserve on a pilot sample.
Documents in several languages — one index or several?
A separate index for a rare language is more accurate than a mixed one. We take a multilingual model when languages are represented comparably, and check on questions where the query language does not match the document language.
What to do with tables and scans?
We export tables in structured form — otherwise rows lose connection with headers. We recognize scans, and a human verifies critical fields: an error in a digit is worse than its absence.
How often to reindex the corpus?
We update incrementally by modification date, full recalculation only when the model or chunking scheme changes. Storage on pgvector updates part of the collection without rebuilding the rest.
How to make sure search finds what is needed?
We collect a reference set of questions with known answers and look at how often the required fragment appears in top positions. Every analyzed failure becomes a new test.
Let's discuss the task: we will return with a plan and estimate for your corpus
A substantive estimate of the scope of work can only come from your data: there are no universal numbers for choosing an embedding model, everything is decided by the corpus and how employees formulate queries.
Send the inputs from the list below — that is enough for us to return with an implementation plan, stages, and composition of a pilot run.
- 20–50 real corpus documents in the formats in which they currently reside: scans, text files, tables, pages of internal knowledge bases.
- Languages and their shares — this determines whether the model can handle it on its own or whether fine-tuning on your vocabulary and abbreviations is needed.
- 20–30 real employee questions with reference fragments: they become the validation set on which we compare candidates.
- Current search pipeline: where documents are stored, how often they are updated, whether access rights are differentiated by section.
- Deployment constraints: your environment or cloud, resource reserve, data storage requirements.
- Operating mode requirements: acceptable response latency, share of manual checks, who and how reindexes the corpus after updates.
Using this data, we select two or three candidates and run them on your corpus: we calculate the share of questions where the required fragment appeared in top retrieval and check robustness to paraphrasing.
The output is an implementation plan with the order of stages, scope of work, and what you will see after each step. Describe the task — we will return with an estimate and a plan.
