Contextual Compression for RAG: Implementation and Optimization

Why Contextual Compression RAG Hits the Context Limit

The classic failure looks like this: a knowledge base of long policies and contracts, a user question, and search returns several dozen fragments.

Only part fits into the model's context window, and the rest is discarded by a formal criterion — often along with the one paragraph that contains the answer.

Externally, the system keeps working: a response comes back, there is a link to the source, but the needed fact is not in it.

The second half of the problem is noise. To avoid losing anything, retrieval depth is increased, and fragments with matching words but a different topic enter the window: similar wording, boilerplate sections, outdated versions of a document.

On long documents, degradation is most noticeable: the model averages several versions of the truth and produces a confident but incorrect answer that is hard to catch during acceptance testing.

Simple trimming of the beginning and end of a fragment saves space, but leaves torn pieces of the paragraph without context.

That is why compression is made semantic: each fragment is reduced to the sentences that actually answer the question, while preserving its link to the original text. More different sources that are on point enter the window, rather than more pieces that resemble one another.

We examine how this is addressed: where to place compression — before or after ranking, how to preserve citations and source verifiability, what to do when compression itself starts discarding relevant content, and which metrics show that the loss of facts has stopped.

The Effect of Context Compression: Accuracy, Speed, Volume

Context compression is not a cosmetic cleanup of the query but a redistribution of the model's attention. Search returns twenty fragments, each only partially relevant to the question, and without compression they go to the model in full, competing with one another. We cut the excess — and only the sentences that actually answer the question remain in the window.

Accuracy: Less Noise — Fewer Errors

Each fragment goes through a second selection and compression: pieces unrelated to the question are cut out, duplicates are collapsed. The share of discarded fragments on typical corpora is 60–85%.

The share of correct answers grows most where the model previously confused documents close in topic: on internal question sets, the gain is 15–25 percentage points.

Speed: Shorter Context — Earlier Answer

The volume of text sent to the model is reduced by 3–5 times. The model processes less data, and response preparation time drops proportionally — this is visible both on individual requests and in batch processing of a document archive.

Volume: Room for Additional Sources

The freed-up space in the window goes to new documents: instead of three, ten fit within the same limit, and the question is resolved without splitting it into several requests.

We assemble the mechanics of selection and reranking on tools such as LangChain — the relevance rules remain yours and are tuned to the corpus, rather than hard-coded into someone else's solution.

Context Compression Modes for RAG: Comparison of Approaches

Context can be compressed at three different steps — before retrieval, within it, and after output — and these modes do not replace but complement one another.

Each offers its own trade-off between answer completeness and processing cost, so the choice should be based on where exactly accuracy is lost: in the query wording, in ranking, or in the volume of fragments sent to the model.

Comparison of Modes

In practice, the modes are combined: threshold filtering cuts obvious noise before expensive reranking, and sentence extraction tightens the context budget at the last step.

The order of application here matters more than the set itself — reranking before filtering is pointless, as it will neatly order what will be discarded anyway.

How We Implement Compression: From Corpus Analysis to Handover

Compression is not "trimming" the retrieved fragments but replacing what enters the model's context in the first place.

Therefore, we start with an inventory of the corpus: where documents are stored, how often they change, which sections carry meaning, and which duplicate one another across versions.

Then the work proceeds step by step, and each step has a verifiable result, not just a configured service.

1. Document corpus analysis. We collect sources, label types (policies, contracts, correspondence, reference guides), and remove duplicates and outdated versions. The output is a corpus map with update frequency for each source.
2. Designing the fragment selection scheme. We determine which units to search by (section, paragraph, clause), how many candidates to retrieve in the first pass, and by what criterion to reject the excess. We fix chunking boundaries so that the answer does not lose meaning at the junction of pieces.
3. Compression setup. We select retrieval by semantic similarity and query reformulation, set the threshold below which a fragment does not enter the context at all, and set a volume limit. Some queries that do not require retrieval are cut off immediately at the input.
4. Building the validation set. We create questions with known correct sources — including cases where the needed answer is spread across several documents and where the question is worded differently from the text.
5. Quality measurement. We calculate the share of queries with the correct source in the context, answer completeness, average volume of transmitted text, and response time. We compare with the baseline variant without compression — separately for each document type.
6. Update policy. We describe how new documents enter the corpus, who confirms the removal of outdated ones, how often the index is rebuilt, and at what measurement deviations retuning is triggered.
7. Handover. We provide configurations, the selection scheme, the validation question set, and measurement instructions so that the team can maintain the solution independently.

The list of steps is not rigid: if the corpus is small and homogeneous, steps 1 and 4 run almost in parallel, and setup takes one cycle. Describe the task — we will return with an estimate and a work plan for your sources.

Implementation Artifacts: Configurations, Schemes, Documentation

After launching context compression in the knowledge base retrieval loop, the client's team retains not only a working service but also a full set of artifacts.

Without them, any change — switching the model, adding a new source, or revisiting thresholds — turns into reverse engineering from code. Below is the list of what we hand over and walk through together with your engineers.

• Selection configuration. Relevance thresholds, deduplication rules, source priority, and context volume limit. It is stored as a separate file — changes are made without rebuilding the service.
• Query processing scheme. The path from the user's question to the answer: parsing, candidate retrieval, compression, context assembly, model invocation — with failure points and filter insertion points.
• Data access rules. Role- and department-based separation: which collections are available to an employee and what is cut before being sent to the model.
• Index update policy. Reindexing frequency, procedure for connecting new documents, and handling of deleted and outdated records.
• Quality check set. Reference questions with expected fragments and hit metrics — a run before every settings change.
• Documentation and policies. Configuration description, request flow diagram, operating instructions, and rollback procedure to the previous settings version.
• Logs and monitoring dashboard. What is written for each request, where to view the failure rate and response time.

We hand over not as an archive but as a walkthrough: we go through each item with your team so that future changes can be made independently.

Case Study: Reduced Context Volume, Accuracy Increased

The client — an internal support assistant that answered from a knowledge base of policies and instructions: retrieval ran on a vector index, and eight retrieved fragments went into the model request.

At the start, this produced a context of about fifteen thousand tokens, a 61% share of correct answers, and seven to eight seconds per response.

The engineers saw a characteristic failure: when a query partially matched a paragraph, the model confidently cited a clause that contained the same words but a different meaning.

The analysis showed that the issue was not the model but the composition of the context — adjacent paragraphs of the document that were unrelated to the question were entering it.

We added a context compression layer based on LangChain: each retrieved fragment goes through an extraction module and leaves only sentences relevant to the query by meaning, not by wording match.

Then the reduced fragments are reranked, and anything below the relevance threshold does not enter the context at all.

Metrics after the improvement. The context volume decreased by 3.4 times — the same eight sources fit into approximately four and a half thousand tokens.

The share of correct answers grew from 61% to 84% on an unchanged test set of 400 questions, and response time fell to three seconds: the model processes less input data and less "noise."

The main result is not speedup but an eliminated class of failures. Answers "inspired by a similar paragraph" disappeared: the system either provides an exact quote from the policy or reports that there is no suitable clause in the base. Such a failure is immediately noticeable to the user and does not undermine trust in the assistant.

How Does Compression During Retrieval Differ from Compression During Answering?

The compression point is the place in the pipeline where a long fragment turns into a short summary.

There are two options: compress immediately after retrieval, when candidates have only been selected by proximity to the query, or later — when the context is assembled for a specific question. This choice determines what exactly we optimize: response speed or answer completeness.

Compression during retrieval processes each retrieved fragment independently and before the final context composition is known. The operation is cheap: it reduces the volume that will then go to the model and lowers processing latency. The downside is that some details are cut blindly, and a necessary caveat may not reach generation.

Compression during answering works with an already selected set and knows the query wording, and sometimes the answer draft. The selection quality here is higher: the summary is built for a specific task, and exactly the connections needed for inference enter the context.

The price is latency: the operation ends up in the critical path of generation rather than in the background.

The main trade-off is accuracy versus volume. The more aggressively we compress, the more compact the context and the faster the answer, but the higher the risk of losing a number, condition, or negation on which the meaning depends.

Too-soft compression returns us to the original problem: an overflowing context, blurred model attention, and increased processing time.

In practice, a two-stage scheme is more often used: a coarse filter at retrieval cuts obvious noise, and a precise one at the answering stage leaves the essentials.

The stages are measured separately — the share of needed fragments in the final context, the latency at each step, and how many answers were spoiled specifically because of compression.

Engineers' Questions: Limitations and Pitfalls of Compression

Compression reduces context volume but does not always improve the answer. Below are the concerns we address before starting: where the approach loses meaning, whether fine-tuning is needed, how it fits on top of current retrieval, and which measurements confirm the effect.

Will the model need fine-tuning for our domain?

Usually not. The base model works confidently on general-language texts, and gaps in your terminology are covered by a domain dictionary, stop words, and rules based on document structure.

Fine-tuning is justified where the corpus has stable jargon and tables that a general compressor cuts along incorrect boundaries.

How will this integrate into already working retrieval?

Compression is a layer after retrieval and reranking, not a replacement: the index, storage scheme, and client requests do not change. A step is added that receives the top fragments and sends a truncated set to the model — in LangChain, this is a separate retriever.

If retrieval is built on another engine, we integrate via its programmatic interface and leave a switch for rollbacks. When a document is updated, only its compressed representations are invalidated — the rest of the corpus remains warm.

How do we know the effect is there?

We run one set of questions with reference sources in two variants — with compression and without. We look at four numbers: the share of answers with a correct source link, average context size, response time, and the share of queries where compression cut out what was needed. We make the decision based on the overall picture, not on volume savings.

Let's Discuss the Task: What Data Is Needed for Evaluation

To calculate the effect of Contextual Compression, we need not access to the production environment but a data slice and the current retrieval settings. Below is the minimum after which we will return with an estimate and a work plan, rather than general considerations.

• Corpus composition: how many documents, average and maximum length, formats (PDF, DOCX, HTML, tables), share of scans and exports from accounting systems.
• Query examples: 50-100 real user formulations with expected answers — we use them to calculate retrieval completeness before and after compression.
• Current retrieval scheme: how the index is built, which fields selection uses, how many candidates are taken as input, and where ranking is applied.
• Context handling: what share of the document goes to the model, how long fragments are truncated, and whether there have been failures due to window overflow.
• Constraints: deployment in an external service or in a closed environment, allowable response time, storage and access rights requirements.
• Quality metrics: how you currently measure the result — hit accuracy, share of "not found" answers, repeat support contacts.

Using this data, we build a measurement on your corpus: we record the current baseline, add compression at the fragment selection stage, and compare. We will break the work plan into stages — data preparation, compression setup, measurement, and documentation handover. Describe the task in free form, and we will clarify the rest ourselves.