Prompt Compression with LLMLingua Tutorial

Long prompts are expensive. A 50,000-token RAG prompt sent 100,000 times a month costs $12,500 at Claude Sonnet 5.5 input pricing. Microsoft Research shipped LLMLingua (and the sequel LongLLMLingua + LLMLingua-2) specifically to shrink prompts without losing information the downstream model needs. In 2026 it is the standard open-source prompt compression library and routinely cuts prompt token count by 2-20x. This article walks through how LLMLingua works, when it is the right fix for your cost problem, and how to wire it into a production RAG or long-context pipeline.

What LLMLingua does

LLMLingua uses a small language model (default: Llama 2 7B or GPT-2) to score the information density of each token in your prompt. Low-information tokens get dropped. The downstream LLM receives a shorter prompt that preserves the semantic content but trims filler words, redundant phrasing, and low-signal context.

Three versions worth knowing:

• LLMLingua (original, 2023): targets 2-5x compression. Works well on structured prompts (few-shot examples, instructions).
• LongLLMLingua (2024): optimized for long-context RAG. Up to 20x compression on 100K+ token contexts.
• LLMLingua-2 (2024): task-aware compression using a trained classifier. Faster than the original, slightly less aggressive.

Install and basic compression

pip install llmlingua

from llmlingua import PromptCompressor

compressor = PromptCompressor(
    model_name="NousResearch/Llama-2-7b-hf",
    device_map="cuda",  # or "cpu" for CPU-only
)

original_prompt = """
You are a helpful customer support assistant. Please answer the user's
question based on the following context.

Context: Our company offers refunds within 30 days of purchase. The item
must be unopened and in its original packaging. Shipping costs are not
refundable. For damaged items, we offer immediate replacement. Our hours
are Monday to Friday 9am to 5pm PST. Call 1-800-555-0100 for support.

Question: How long do I have to return a product?
"""

result = compressor.compress_prompt(
    context=[original_prompt],
    ratio=0.5,  # Keep 50% of tokens
)
print(result["compressed_prompt"])
print(f"Original: {result['origin_tokens']} tokens")
print(f"Compressed: {result['compressed_tokens']} tokens")
print(f"Rate: {result['rate']}")

Typical result: 150-token prompt compresses to 60-80 tokens, saving 40-60% of input cost while preserving the answer-relevant information.

LongLLMLingua for RAG pipelines

RAG prompts are the canonical use case: you retrieve 10 chunks of 500 tokens each (5,000 total context tokens), the user question is 50 tokens, system prompt 200 tokens. Total: 5,250 tokens per request. LongLLMLingua can cut the RAG context 10-15x while preserving answer accuracy on standard benchmarks.

compressor = PromptCompressor()

retrieved_chunks = [chunk1_text, chunk2_text, chunk3_text, ...]
question = "What is our refund policy?"

compressed = compressor.compress_prompt(
    context=retrieved_chunks,
    instruction="Answer the question based on the context.",
    question=question,
    target_token=500,  # compress context to 500 tokens
    rank_method="longllmlingua",
    condition_in_question="after",  # question-aware compression
    reorder_context="sort",  # reorder chunks by relevance
)

# compressed["compressed_prompt"] is now ~500 tokens instead of 5000
response = openai_call(compressed["compressed_prompt"])

The condition_in_question parameter is the key difference from the base version: compression keeps tokens most relevant to the specific question, not just high-information tokens generically. Preserves answer accuracy at aggressive compression ratios.

LLMLingua-2: faster, classifier-based

LLMLingua-2 trains a small BERT-style classifier to score each token for “preservation worthy”. Runs 3-6x faster than the original (which uses a 7B LLM for scoring). Less aggressive compression but much lower overhead per request.

from llmlingua import PromptCompressor

compressor = PromptCompressor(
    model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
    use_llmlingua2=True,
    device_map="cuda",
)

compressed = compressor.compress_prompt(
    original_prompt,
    rate=0.4,
    force_tokens=["\n", ".", "!", "?", ","],
)

Latency: 10-50ms on GPU, 100-300ms on CPU. LLMLingua-2 is practical for request-time compression in a hot path; the original LLMLingua is usually better suited for offline/pre-compute pipelines.

Economic math: when it is worth it

Rough breakeven analysis:

• GPU rental for LLMLingua: ~$0.50/hour for a modest GPU (A10 or T4) handling the compressor
• Compute per request: 50-200ms on GPU, call it $0.00003 per request amortized
• Token savings per request: 2-10x reduction on a 5K-token prompt = $0.015-$0.045 saved at GPT-4o pricing

At 100K requests/month:

• Infrastructure: $360/month GPU + $3 compute = $363
• Savings: 100K * $0.025 avg = $2,500/month
• Net: $2,137/month saved

At under 10K requests/month, the GPU rental usually swamps the token savings. Use LLMLingua-2 on CPU (free) or just skip compression and let shorter prompts pass through.

Where compression hurts accuracy

Not every prompt tolerates compression well:

• Structured data (JSON, tables, code): dropping tokens corrupts structure. Compression breaks parse. Skip compression on structured inputs.
• Precise facts (numbers, dates, names): compression may drop a specific number or name the answer depends on. Fine for RAG context, less fine for financial or medical data.
• Few-shot examples with specific format: compression can strip format cues, making the model’s output format drift. Mark few-shot examples with force_tokens to preserve them.
• Short prompts (under 500 tokens): compression overhead exceeds the savings. Skip under 500 tokens; apply only on long contexts.

Always measure accuracy before/after compression on your evaluation set. Expect 2-5% quality drop at 50% compression on well-structured prompts; up to 10% drop at 90% compression. If quality hit is unacceptable, lower the compression ratio.

Combining with other cost tactics

Prompt compression stacks with other cost optimizations:

• + Semantic caching: cache the UNcompressed input-to-output mapping. Serve cached responses when similar questions come in. Compression only kicks in on cache misses.
• + Prompt caching (OpenAI/Anthropic native): compress once, cache the compressed version in the model provider’s cache. First call pays compression + full input cost, subsequent calls get cached-input pricing.
• + Model routing: compress for the expensive model (GPT-5, Opus 5.5), skip compression for cheap models (Haiku 5.5, GPT-4o-mini) where cost savings do not justify quality risk.
• + Batch API: compress + batch for maximum savings on offline processing. 50% off from Batch + 60% token reduction from compression = ~80% total cost cut.

Production deployment patterns

Three common shapes:

1. Request-time compression: LLMLingua-2 as a middleware in your API handler. Adds 10-50ms latency, immediate savings. Best for user-facing chat.
2. Offline pre-compression: compress your RAG document chunks once at ingestion, store the compressed versions in the vector DB. Zero per-request latency. Best for stable RAG corpora.
3. Hybrid: pre-compressed static context (docs, system prompt) + request-time compression for the user’s dynamic input. Covers both axes.

Frequently Asked Questions

Angel Jude Suarez

Full-Stack Developer at PIES IT Solution

Focuses on Python development, machine learning, and AI integration. Has built production AI systems including OpenAI Whisper integration for medical transcription and GPT-4o-powered diagnosis assistance. Strong background in pandas, scikit-learn, and TensorFlow.

Expertise: Python · PHP · Java · VB.NET · ASP.NET · Machine Learning · AI Integration · OpenCV · Django · CodeIgniter  · View all posts by Angel Jude Suarez →