Prompt compression, retrieval filtering, session compaction, caching, and agent context optimization all aim to make the context a model processes cheaper. They work in very different ways, they fail in different ways, and they are often confused.
This guide explains each method, what it changes, where it falls short, and which tools and research document it today. It is written by PointFive, the maker of TokenShift. It is an evaluation framework with documented examples, not an independent ranking or a benchmark of competing products: tools are listed as examples of a method, in no order.
TLDR
- Caching is not compression. Provider caching keeps the same context and bills repeated parts at a lower rate. It is usually the first thing to check.
- Token-level compression (LLMLingua and LLMLingua-2, The Token Company, Kong's prompt compressor) deletes low-value tokens from a prompt. Savings and quality depend heavily on the task.
- Retrieval filtering and reranking (LangChain and LlamaIndex compressors, Cohere Rerank) decide which documents enter a prompt at all.
- Compaction (Anthropic and OpenAI) summarizes or clears older turns in long agent sessions.
- Research methods (Selective Context, RECOMP, Gisting, AutoCompressors, 500xCompressor) show what is possible; several require training or model access that closed API models do not allow.
- Agent context optimization acts on the context agents assemble while they work.
- Measure completed-task quality and total cost, not the percentage of tokens removed.
Choose the mechanism before the product
| Method | What it changes | What it saves | Main risk |
|---|---|---|---|
| Provider caching | Billing for repeated context | Price of repeated input | Only helps with stable prefixes; writes can cost more |
| Token-level compression | The prompt text | Input tokens | Removing information can change answers |
| Retrieval filtering and reranking | Which documents are included | Input tokens in RAG | Dropping a document that mattered |
| Compaction and summarization | Older turns in a session | Input tokens in long sessions | Losing requirements, decisions, or tool results |
| Trained compression methods | The model's view of the context | Input tokens and compute | Requires training; not usable with closed APIs |
| Agent context optimization | Context assembled by agents | Agent input tokens | Removing context the agent still needs |
1. Provider caching: keep the context, pay less for it
Provider caching reuses eligible context. It changes billing and processing behavior without necessarily reducing the token count shown in an application.
- Anthropic: prompt caching documents separate cache writes and reads. As of October 7, 2026, reads cost 0.1x the base input price for most models (lower for some newer models), and writes cost 1.25x for the default 5-minute lifetime or 2x for 1 hour.
- Google: the Gemini API distinguishes implicit and explicit caching. Implicit caching is on by default for Gemini 2.5 and newer models, with no guaranteed saving; explicit caching guarantees a reduced rate but adds a storage charge for as long as the cache is kept. Gemini Enterprise Agent Platform (formerly Vertex AI) documents the same distinction.
Eligibility, model, prefix matching, lifetime, and traffic patterns matter. Caching is not universally free or a guaranteed saving. For other providers, see the token optimization tools guide.
Interaction with compression: compressing a prompt changes its text, which can break a cached prefix. A compressed prompt that misses the cache can cost more than an uncompressed prompt that hits it. Evaluate the two together.
2. Token-level compression
Token-level compression uses a smaller model to score each token and remove those least likely to matter, then sends the shorter prompt to the target model.
- LLMLingua (Microsoft Research, MIT license) is a family of methods:
- LLMLingua reports up to 20x compression with little performance loss, on reasoning and conversation benchmarks.
- LongLLMLingua is question-aware and aimed at long-context and retrieval scenarios. It reports improved accuracy with around 4x fewer tokens on NaturalQuestions with GPT-3.5-Turbo.
- LLMLingua-2 is a task-agnostic token classifier that reports 3x-6x faster compression than earlier methods, at compression ratios of 2x-5x.
The repository is not archived, but the latest package release (0.2.2) is from April 2024. Integrations exist for LangChain, LlamaIndex, and Prompt flow.
- The Token Company offers compression as a hosted API and SDK: a classifier scores each token and deletes the ones least likely to affect the answer. Its published accuracy and speed results are vendor-reported. It is an early-stage company (Y Combinator, Winter 2026).
- Kong AI Prompt Compressor is an Enterprise plugin for Kong AI Gateway 3.11 and later, built on LLMLingua-2. It compresses to a target ratio or token count, and requires a separate compressor service.
What it saves: input tokens on long prompts. What to watch: published ratios come from specific models and datasets. The best case on one benchmark is not a typical result on your workload. Compression also adds a step to every request; account for its latency and cost.
3. Retrieval filtering and reranking
In retrieval-augmented generation (RAG), the cheapest token is the one you never retrieve. Filtering and reranking select which passages enter the prompt.
- LangChain: ContextualCompressionRetriever wraps a retriever and compresses its results, with document compressors that filter by embedding similarity, extract only relevant statements with an LLM, or rerank. In LangChain v1 these moved to the
langchain-classicpackage; check your version before building on them. - LlamaIndex: node postprocessors such as SentenceEmbeddingOptimizer remove sentences that are not relevant to the query, and a LongLLMLingua postprocessor applies LLMLingua compression to retrieved nodes.
- Cohere Rerank orders documents from most to least relevant to a query, so you can keep only the top results.
What it saves: input tokens in RAG, often with better answers, because irrelevant passages are a source of distraction as well as cost. What to watch: assess whether the remaining evidence supports the answer, including uncommon but important documents. A filter tuned on common questions can drop the one passage a rare question needs.
4. Compaction and summarization for long sessions
Agents and chat sessions grow with every turn. Compaction replaces or clears older context so a session can continue without re-sending everything.
- Anthropic documents compaction, which replaces older turns with a summary written on the server, and context editing, which clears old tool results and thinking blocks. Both are in beta. Compaction usage is reported separately from the top-level usage figures, so include it when you measure cost.
- OpenAI documents compaction in the Responses API, triggered at a token threshold or through a standalone endpoint. The compacted item is encrypted and not human-readable, so you cannot inspect what was kept.
- Conversation summarization in your own application does the same job with your own prompt and model.
What it saves: input tokens on every later turn of a long session. What to watch: check whether requirements, decisions, and tool results survive the transformation. A summary that drops a constraint can cause the agent to redo or undo work.
5. Research methods: what is possible, and what it requires
Academic work explores more aggressive compression. Several of these methods require training or access to model internals, which closed API models do not provide.
- Selective Context removes redundant content using a small model's measure of information. It reports a 50% reduction in context cost with a small quality drop on summarization, question answering, and response generation. Its repository has not been updated since early 2024.
- RECOMP trains extractive and abstractive compressors that summarize retrieved documents before they enter the prompt, and can return nothing when documents are irrelevant. It reports compression rates as low as 6% with minimal performance loss on language modeling and open-domain QA.
- Gisting trains a model to compress a prompt into a few cached "gist" tokens, reporting up to 26x prompt compression. It requires training the model itself.
- AutoCompressors and 500xCompressor compress context into learned vectors or special tokens. 500xCompressor reports retaining roughly 62% to 73% of the model's capabilities at its compression ratios, a reminder that extreme compression has a visible cost.
What to take from them: compression ratio and answer quality trade off against each other, and the strongest ratios come from methods that change the model. For teams calling closed models through an API, text-level methods (sections 2 to 4) are the practical options.
6. Agent context optimization
Agent context optimization considers the context assembled during an agent's work: files read, tool output, history, and instructions.
TokenShift is the AI agent control plane for your employee workstations. It records which tool and MCP server ran, each failure and retry, and how long it took, and suggests better ways to work with agents. Data is processed on the device. Smart Routing is a separate opt-in module that can select a different model.
For how agents accumulate unnecessary context, see our research on prompt-induced waste in coding agents.
Measure successful work, not just smaller prompts
Use representative tasks, including failures and long sessions. Compare the baseline with each intervention using the same acceptance criteria. Record input, output, cache-read and cache-write usage, tool charges, optimization overhead, retries, latency, and completed-task quality.
Do not multiply a token reduction percentage by the whole bill. For an illustrative calculation, reducing 3 billion uncached input tokens fivefold removes 2.4 billion tokens. At an assumed input price of $15 per million, that is $36,000 in gross input charges, before any changes to caching, outputs, retries, or optimizer cost. This is arithmetic under stated assumptions, not a current price quote or a measured saving.
Fewer tokens is not the same as lower cost. See our research on why token reduction is not cost reduction.
Treat long-context research as evidence with boundaries
Lost in the Middle examined information position in question-answering and retrieval tasks. Its findings do not establish a universal accuracy decline for longer prompts or guarantee that compression improves an agent's work. Test your model, context, and task directly.
Questions to ask before adoption
- Which data leaves the execution environment, and where is it retained?
- Does the tool remove context, reuse a cache, choose another model, or combine these actions?
- Which agents and runtimes are supported in the version being evaluated?
- Are reported savings net of retries, cache changes, and the tool's own cost?
- Were published results measured on models and tasks like yours?
- Is the project maintained, and who owns it?
- Can the team disable the intervention and compare completed-task outcomes?
Frequently asked questions
What is prompt compression?
Prompt compression reduces the number of tokens sent to a model while trying to keep the information the model needs, by deleting low-value tokens, filtering retrieved documents, or summarizing earlier context.
Does prompt compression guarantee lower cost?
No. Savings depend on billable token categories, caching, output behavior, retries, and the cost of the intervention.
Is caching the same as compression?
No. Caching reuses eligible context; compression changes the context supplied to the model. They can interact and should be evaluated together.
Does compression reduce answer quality?
It can. Removing information can change answers, and published results apply to the models and datasets tested. Measure completed-task quality on your own workload before and after.
Which method should I try first?
Check provider caching first, then reduce what you retrieve and how long sessions grow. Token-level compression is worth testing on long prompts where those steps are not enough.
Methodology
Capabilities, results, and maintenance status were taken from official documentation, repositories, and the original papers, linked above, and checked on October 7, 2026. Reported results are quoted with the conditions under which they were measured; they are not guarantees for other workloads. Tools are listed as examples of each method, not ranked.