Chunking strategies for RAG that actually work

Most "RAG isn't working" problems are chunking problems wearing a costume. The model looks dumb, the answers are vague or wrong, and everyone blames the LLM — but the real failure happened upstream, when you sliced your documents into pieces and the piece holding the answer either never got retrieved or arrived stripped of the context that made it mean anything. Chunking is the least glamorous part of a retrieval system and the one that most decides whether it works.

I build answer engines that have to ground every response in specific source material, so this is the trade-off as I actually make it, not a tour of libraries. If you want the wider picture of when retrieval is even the right tool, that's RAG vs fine-tuning vs long context. This piece is one layer down: how to cut the documents so the retriever can find what it needs.

Why chunking makes or breaks retrieval

Retrieval works by embedding each chunk into a vector, embedding the query the same way, and returning the chunks whose vectors sit closest. Two things follow from that mechanic, and they pull against each other.

First, a chunk is the smallest unit you can retrieve. If the answer is spread across a chunk boundary, no single chunk contains it, and no amount of clever prompting recovers what retrieval never fetched. Second, an embedding is an average of everything in the chunk. Stuff ten topics into one big chunk and its vector becomes a blurry centroid that's not especially close to any specific question — so the chunk that literally contains your answer scores lower than a tighter chunk that's merely on-topic. Chunk too big and you dilute; chunk too small and you fragment. Every strategy below is a different answer to that tension.

The strategies, worst to best

Fixed-size cuts every N tokens or characters and moves on. It's fast, dead simple, and completely blind to meaning — it will split a sentence mid-clause and cut a definition away from its term. Fine as a baseline; rarely what you want to ship.

Sentence or paragraph splitting respects natural language boundaries, so chunks at least start and end at real breaks. Better, but paragraph lengths vary wildly — one is a heading, the next is a wall of text — so you get chunks of wildly uneven size, which makes them behave inconsistently at retrieval time.

Recursive chunking is the pragmatic default and what I reach for first. It splits on a priority list of separators — paragraphs, then sentences, then words — falling down the list only when a piece is still too big for your target size. You get chunks that honor structure and stay within a size band. For most prose corpora this is the sweet spot: most of the quality of something fancier, almost none of the cost.

Structure-aware / markdown-aware chunking uses the document's own scaffolding — Markdown headings, HTML sections, code structure — as the split points, and carries the heading path into each chunk so a fragment still knows it lives under "Returns > International > Exceptions." If your content has real structure, use it; it's the single highest-leverage upgrade over fixed-size.

Semantic chunking embeds sentences and splits where the topic actually shifts, so each chunk is one coherent idea. It's the highest quality on messy, unstructured text and the most expensive to build, since you're running embeddings just to decide the cuts. Reach for it when recursive chunking demonstrably isn't separating topics cleanly — not by default.

A quick way to hold the whole ladder in your head: fixed-size ignores meaning, sentence and paragraph respect language, recursive respects size and structure, markdown-aware respects the document's own map, and semantic respects meaning directly. You climb the ladder only when the rung below it stops retrieving the right passage — each step up costs more to build and maintain, so don't pay for semantic when recursive already lands the answer in the top results.

Size and overlap: rules of thumb

Start at 200–500 tokens per chunk with 10–20% overlap. That's a starting line, not an answer. The right size tracks how your documents carry meaning: dense reference material and FAQs want smaller, precise chunks; flowing narrative or argument wants larger ones so the thread survives.

Overlap means each chunk repeats a little of its neighbors' text, so a fact sitting on a boundary lands whole in at least one chunk instead of being sliced in two. But overlap is a patch for bad boundaries. If you split on real structure, a fact rarely straddles a split, and you need far less of it. Crank overlap too high and you flood the index with near-duplicate chunks that compete with each other for the top-k slots and crowd out genuine variety. Treat 15% as a default and structure as the real fix.

Metadata and parent-document retrieval

Every chunk should carry metadata: source document, section heading, page, date, type. That buys you two things — the model sees where a passage came from so it can cite it, and you can filter retrieval ("only this product's docs," "only content newer than X") before ranking even runs. This is a core context-engineering discipline; the chunk isn't just text, it's text plus provenance. More on assembling that working set in context engineering.

The pattern that resolves the small-vs-large dilemma is parent-document (or retrieval-window) retrieval: index small, precise child chunks so matching is sharp, but at fetch time return the larger parent section — or the chunks immediately around the hit — so the model gets the full context. You retrieve on the tight vector and read on the wide passage. It's the closest thing to having your cake and eating it, and it's usually worth the extra plumbing.

Retrieval quality isn't chunking alone — how you match matters too. Dense vectors miss exact terms like SKUs and error codes that keyword search nails, which is why hybrid setups win; that's vector search vs keyword search.

Tables, code, and PDFs

Generic splitters mangle non-prose, and this is where fixed-size chunking does the most damage.

• Tables: keep the header row with its data, or serialize each row into a self-describing line ("Region: EU, Ship time: 5 days") so a retrieved fragment is still readable without the rest of the grid.
• Code: split on functions and classes, never line counts — retrieving half a function is useless.
• PDFs: the hard part is before chunking. Extraction scrambles multi-column layouts, headers, and footers into nonsense, and no chunking strategy saves garbage input. Fix extraction first, chunk on the recovered structure second.

How to evaluate chunking against real queries

Don't argue about chunk size — measure it. Build a small evaluation set: real questions, each paired with the passage that should answer it. Twenty to fifty pairs drawn from questions your users actually ask is enough to see a config move; you don't need a benchmark, you need ground truth for your corpus. Then track recall at k — did the correct passage land in the top k results — because that's the number chunking most directly moves. Answer quality depends on the model too, but if the right chunk never gets retrieved, no model can save the answer, so recall at k isolates the part chunking owns.

Change one variable at a time (size, then overlap, then strategy), re-run, keep what retrieves the right passage most often. Resist the urge to tune three knobs at once — you'll never know which one helped, and next month's regression will be a mystery. This is the same discipline as any other part of the stack; see evals: know your AI works. And if you want the whole system this sits inside, it's the building with LLMs field guide.

The takeaway: chunking isn't preprocessing you do once and forget. It's a retrieval decision you tune against real queries, and it's usually the cheapest, highest-leverage fix when your RAG system is quietly returning the wrong thing.