The Chunking Bet: How One Early Decision Haunts Every Query

A mistake everyone makes

If you built a RAG system in the last two years, you probably made the same move everyone else made. You split your documents into 512-token chunks. You generated embeddings. You stored them in a vector database. You called it done.

For a while it worked. The system answered questions. Stakeholders were happy. You moved to the next feature.

Then something strange started happening. Simple questions got complex answers. Complex questions got incomplete ones. Users said the system "forgot" context that was clearly sitting in the source documents. You checked the logs and found retrieval working exactly as designed: it returned the most semantically similar chunks. Those just were not the right chunks.

This is the chunking dilemma. Nobody talks about it at conferences. It is costing companies real money in lost productivity and frustrated users.

Why this matters now

The RAG market grew to $1.85 billion in 2024. It is expanding at a 49% compound annual growth rate. By 2027, 70% of enterprises are projected to run RAG in production.[11,12] Millions of developers are building on one flawed assumption: that a single chunking strategy can serve every purpose.

Here is what that looks like at scale. You start with fixed-size chunking, say 512 tokens with 50 tokens of overlap. It works for your first use case: answering factual questions about product documentation. Then the business adds financial reports. Then legal contracts. Then support transcripts. Then internal wikis. Each new data type has its own structure, its own information density, its own typical questions.

You are stuck with your original strategy. Changing it means re-embedding the whole corpus, and that corpus has grown from 1,000 documents to 100,000. The compute cost is high. The engineering effort is bigger. So you do not change it. You keep adding new data to a system tuned for data that no longer represents most of your use case.

There is a way to make chunking decisions at query time instead of index time, one that lets the system adjust based on what a user is actually asking for. That is the promise behind Retrieval Language Models, the successor idea to traditional RAG. Before we get to the fix, we need to see how deep this problem actually goes.

The bet you make at index time

Traditional RAG makes a critical trade at the moment of indexing. It chunks your documents once, embeds them once, and stores those embeddings. That buys you speed, scale, and predictability. It costs you flexibility. The chunking strategy freezes in time, tuned for no specific query and therefore optimal for none.

Picture two questions. "What were the key financial risks identified in Q3?" needs a chunk of maybe 1,024 tokens to hold a complete risk narrative. "What was the revenue in Q3?" needs something closer to 128 tokens, a single table cell with context. Your fixed 512-token chunks serve neither one well.

NVIDIA ran a 2024 benchmark testing nine chunking strategies across multiple document types.[8] The gap between the best and worst approach hit 9 percentage points in recall. That is the line between a system people trust and one they give up on.

The "best" strategy also moved with the query type. Factoid questions (who, what, when, where) did best with small chunks, 256 to 512 tokens. Multi-step analytical questions needed large chunks, 1,024 tokens or more. Summaries landed in between. Legal documents wanted page-level chunks. Chat transcripts needed conversational turn boundaries. Medical records needed semantic section breaks that kept clinical context intact.

Users do not label their questions "factoid" or "analytical." They just ask. Your system has to guess the right chunking strategy in real time from whatever thin signal sits in the query text.

The coherence problem

There is a second layer under all this: semantic coherence. Split a document at an arbitrary token boundary and you often cut through a semantic unit. A 512-token chunk can start mid-sentence, or worse, mid-argument. The start lacks context. The end is incomplete. The embedding represents a fragment, not a coherent piece of meaning.

Researchers at Jina AI call this the late chunking problem.[4] Chunk before you embed, and you cap the quality of the semantic representation from the start. The embedding model sees isolated fragments instead of contextualized passages. Their fix, embedding first over longer context and chunking after, improved retrieval accuracy. It also meant rethinking the whole pipeline.

Nine ways to cut a document

Practitioners have gone well past fixed-size chunking. By 2025, production RAG systems use at least nine distinct strategies.[1,6,7]

Fixed-size chunking. The baseline. Split every N tokens, usually 256 to 512, with some overlap. Fast, simple, and bad at preserving meaning. Most systems still run this because it's what most tutorials teach.

Recursive character splitting. A step up. It tries to split on natural boundaries like paragraphs and sentences. Still doesn't understand content. A paragraph break in the middle of a complex argument is still a bad place to cut.

Semantic chunking. Uses embedding similarity between consecutive sentences to find good split points. When similarity drops, insert a boundary. Smart in theory. Expensive to run and prone to over-segmenting.

LLM-based chunking. A language model finds the logical breaks. You prompt it: "Where would you naturally divide this document?" Highest quality, slowest, most expensive. Works for one-time processing or high-value documents, not much else.

Agentic chunking. AI-driven, dynamic, with metadata added at each cut. The model doesn't just find breakpoints, it writes a summary for each chunk: "This section covers Q3 revenue in the context of year-over-year growth." Powerful. Complex to run.

Late chunking. Embed over long context first, then chunk while keeping the full-context embeddings. Fixes the coherence problem. Needs specialized embedding models with long context windows.

Contextual chunking. Anthropic's approach: use Claude to write a short context description for each chunk.[13,14] Each chunk gets a preamble, something like "This section appears in a financial report discussing..." The embedding covers both the chunk and its context. Effective. Not cheap.

Proposition-based chunking. Pull out atomic, claim-level statements instead of chunking by document structure. "Q3 revenue was $50M" becomes its own chunk regardless of where it sits in the document. Strong for factual retrieval. Weak for complex reasoning.

Adaptive or hierarchical chunking. Chunk size adjusts to content structure. Dense technical sections get small chunks. Narrative sections get large ones. The most sophisticated option, and the hardest to build.

Every one of these is a compromise. Each optimizes for a slice of query patterns at the cost of the rest. And once you pick one and index your corpus, you are locked in. Changing strategies means re-embedding everything.

That re-embedding cost, and what it actually does to a team running one of these systems in production, is where we pick up next.

Continue to Part 2: The performance data, the hidden cost of re-embedding, and the cognitive load problem →

Works Cited

[1] "Chunking Strategies for RAG: A Comprehensive Guide." Medium, November 2024. [4] Günther, M., et al. "Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models." Jina AI, 2024. [6] "The Rise and Evolution of RAG in 2024." RAGFlow, December 2024. [7] "Best Chunking Strategies for RAG in 2025." Firecrawl, 2025. [8] "RAG Evaluation: Best Practices and Common Pitfalls." NVIDIA Technical Blog, 2024. [11] "Retrieval-Augmented Generation Market Analysis and Forecast 2024-2027." Industry Market Research, 2024. [12] "Enterprise AI Adoption: RAG Integration Trends." Technology Research Report, 2024. [13] "Introducing Contextual Retrieval." Anthropic, 2024. https://www.anthropic.com/news/contextual-retrieval [14] "Contextual Chunk Embeddings: Best Practices for RAG Systems." Anthropic Technical Documentation, 2024.