Document Chunking Strategies for RAG: Fixed vs Semantic vs Agentic (2026)

Chunking is where most RAG pipelines quietly fail. The retriever is only as good as the chunks you give it, and the chunks are produced by a splitter that gets written once and never revisited. By the time retrieval quality shows up as a problem, the fix usually starts at the chunker, not the embedding model.

This guide covers the three chunking families in use today, when each one wins, and the measurements that tell you which to pick for a given document set. The right strategy is almost always obvious once you look at the actual documents, which is where most teams skip ahead.

Why chunking matters more than the embedding model

Teams spend weeks picking between OpenAI, Cohere, and Voyage embeddings. The quality gap between those three is 2-5% on most benchmarks. The gap between fixed-size and semantic chunking on the same documents is 20-40%. Chunking strategy is the lever with the biggest impact on retrieval quality, and it costs nothing to switch.

Three things a chunker has to balance:

• Topical coherence. Everything inside one chunk should talk about one thing. A chunk that spans two topics gets a muddy embedding that scores poorly on both.
• Enough context. A single sentence often does not carry the meaning. Chunks need enough surrounding text that the model can answer from the chunk alone.
• Retrievable size. Smaller chunks are more precise but you need more of them in the context window. Larger chunks waste tokens on irrelevant text.

No chunking strategy wins all three. Pick the one that trades favorably for your actual document shape.

Fixed-size chunking: the default

Split the document into equal-size pieces, usually 200 to 800 tokens, with 10 to 20% overlap. The simplest possible strategy, and it works on most document sets.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " "],
)

chunks = splitter.split_text(long_document)

The separator list is the quiet upgrade. Without it, you cut mid-sentence and the chunk ends with a dangling clause. The RecursiveCharacterTextSplitter tries to split on paragraph breaks first, then line breaks, then sentences, then words. The chunk ends on a natural boundary when one is available.

Fixed-size works well on:

• FAQ pages and knowledge base articles (uniform paragraph lengths)
• Product documentation
• Email threads and chat transcripts
• News articles

It fails on:

• Legal contracts where a clause mid-page is semantically different from the clauses around it
• Research papers with sections of wildly different lengths
• Code files, where cutting mid-function destroys meaning
• Tables and structured data

Semantic chunking: split where meaning changes

Semantic chunking looks for topic shifts and splits there. Three ways people do it in 2026:

Heading-based. Split on Markdown headings or HTML heading tags. The chunks match the author’s own structure. Works beautifully on technical docs, policies, and anything written with structure.

Embedding-similarity-based. Compute embeddings of consecutive sentences. Where cosine similarity between two neighbors drops below a threshold, insert a chunk boundary.

from llama_index.core.node_parser import SemanticSplitterNodeParser
from llama_index.embeddings.openai import OpenAIEmbedding

splitter = SemanticSplitterNodeParser(
    buffer_size=1,
    breakpoint_percentile_threshold=95,
    embed_model=OpenAIEmbedding(model="text-embedding-3-small"),
)

nodes = splitter.get_nodes_from_documents(documents)

Embedding-similarity chunking costs more at index time (one embedding call per sentence) but it produces tighter, more coherent chunks on prose where authors do not write with explicit section breaks.

Discourse-marker-based. Split on transition phrases like “First,” “In conclusion,” “On the other hand.” Lightweight and useful on transcripts of meetings, podcasts, lectures, or interviews where speakers rarely signal topic shifts with headings.

Agentic chunking: let the model decide

Agentic chunking uses an LLM to read the document and decide where chunks should start and end. The model gets instructions like:

Split this document into chunks where each chunk:
- Covers exactly one topic or sub-topic
- Is self-contained, with any references resolved
- Is between 300 and 800 tokens
- Includes a one-sentence title you generate

Return a JSON list of {title, start_line, end_line, summary}.

Agentic chunking produces the best results on three document types: legal documents (where clause boundaries matter more than paragraph boundaries), long-form research (where arguments span pages), and multi-topic reports (annual reports, consulting decks).

The cost is real. At $3-15 per million input tokens, chunking a 100-page document with Claude Sonnet 5.5 costs $0.50 to $2.00. For a 10,000-document corpus that is $5,000 to $20,000 of one-time indexing cost. Only use agentic chunking when the quality gap vs semantic chunking is worth the money, which you should measure before committing.

Chunk size: what the research says

Across the LangChain, LlamaIndex, and Pinecone benchmarks published 2024-2026, three patterns hold:

• Short questions want short chunks. If the typical query is 5-15 words (“What is X?”), 200-400 token chunks retrieve better.
• Analytical questions want longer chunks. If the typical query asks for reasoning across multiple facts (“Compare X and Y”), 600-1,000 token chunks give the model enough context.
• Overlap only helps up to about 20%. Overlap above 25% buys nothing but larger indexes and higher retrieval cost. Below 10%, you risk cutting a sentence off from its referent.

The practical starting range is 500 tokens with 10% overlap. Measure retrieval quality, then move up or down based on what your query distribution actually looks like.

Chunk expansion with parent-child indexing

A newer pattern that avoids the small-vs-large tradeoff: index small chunks for retrieval precision, but return their parent (a larger surrounding window) to the generator.

from llama_index.core.node_parser import HierarchicalNodeParser

parser = HierarchicalNodeParser.from_defaults(
    chunk_sizes=[2048, 512, 128],
)

nodes = parser.get_nodes_from_documents(documents)

LlamaIndex builds a tree: 128-token leaves for retrieval, 512-token parents for context expansion, 2048-token root for whole-section fallback. The retriever matches at the leaf level and returns the parent window. Precision stays high because leaves are focused. Generation quality stays high because the model sees enough surrounding text to understand.

Parent-child indexing is close to free at query time. It doubles or triples the index storage, which rarely matters.

Chunking tables, code, and PDFs

Three special-case rules that trip up most pipelines:

Tables. Never split a table mid-row. Convert each table to Markdown and keep the full table as one chunk, with a short lead-in sentence describing what the table shows. For large tables, split row-wise but repeat the header on every chunk.

Code. Split on function or class boundaries, not fixed sizes. Tree-sitter plus LangChain’s Language.PYTHON splitter does this cleanly for most languages. Include the function signature at the top of every chunk.

PDFs with layout. Pass through a layout-aware parser (Unstructured, pymupdf, pdfplumber) before chunking. Column text, headers, footers, and footnotes need to be separated from the body or they contaminate every chunk.

These rules account for about 60% of the “my RAG system does not work” tickets. The chunker was happy and emitted chunks; the chunks were just garbage because the document structure was never respected.

How to measure chunk quality

Pick 20 to 50 question-answer pairs that cover the range of real queries. For each one, record whether the correct chunk is in the top-5 retrieved. Score = correct retrievals / total queries.

Three measurements worth tracking:

1. Top-5 recall. Does the right chunk appear in the top-5 at all?
2. Mean reciprocal rank. How high does the right chunk rank? Weights top-1 heavier than top-5.
3. Chunk coherence. Pick 10 random chunks and have a human (or a model) grade whether each chunk is topically coherent.

Run the measurement after any chunker change. Teams that measure typically converge in 2-3 iterations. Teams that eyeball retrievals and ship often re-write their RAG pipeline from scratch at month 6.

Decision tree for picking a strategy

Six questions, in order:

1. Does the document have explicit headings or structure? If yes, use heading-based semantic chunking.
2. Is it code? Use language-aware splitting.
3. Is it tables or structured data? Keep tables whole, chunk around them.
4. Is it long-form prose without structure (transcripts, essays, novels)? Use embedding-similarity semantic chunking.
5. Is it legal or research where argument structure matters more than paragraphs? Agentic chunking, budget permitting.
6. None of the above? Fixed-size 500 tokens with 10% overlap is a defensible default.

Add parent-child indexing on top of whichever strategy wins. It is almost free and usually helps.

Frequently asked questions

Pick the simplest strategy that covers your document shape. Measure before you optimize. Add parent-child indexing on top of whatever wins. That gets most teams within 5-10% of the retrieval quality that over-engineered pipelines deliver, at a fraction of the complexity.

Joken E. Villanueva

Founder & Lead Developer at PIES IT Solution

Founder of PIES Information Technology Solutions, a software company building production-grade applications for institutions across the Philippines. Over 8 years of hands-on full-stack development experience, currently leading the development of ClinicAI, an AI-powered clinic management platform.

Expertise: PHP · MySQL · JavaScript · AI Integration · SaaS Architecture · VB.NET · Database Design · Capstone Documentation · Java  · View all posts by Joken E. Villanueva →