Overview
RAG retrieves only the knowledge-base chunks relevant to a specific query at request time, rather than fitting an entire knowledge base into every prompt, which both respects the context window and keeps the model focused on genuinely relevant background instead of diluting accuracy with mostly-irrelevant content. The part that most naive RAG implementations get wrong is chunking: a chunk pulled out of its surrounding document, "the company's revenue grew by 3% over the previous quarter," reads as generic and context-free once isolated, which company, which quarter, and that missing context means its embedding won't reliably match the queries it should. Anthropic's published contextual retrieval approach fixes this by prepending a short, automatically-generated explanatory context to each chunk before embedding it, and measured a 49% reduction in retrieval failures from that change alone, 67% combined with reranking.
Quick Reference
| Stage | What happens |
|---|---|
| Chunking | Documents split into smaller pieces, since the full knowledge base won't fit the context window |
| Embedding | Each chunk converted to a vector and stored in a searchable index |
| Retrieval | At query time, the most relevant chunks are found via similarity search |
| Generation | Retrieved chunks are added to the prompt as context for the model's response |
| Approach | Retrieval failure reduction (Anthropic's published results) |
|---|---|
| Contextual retrieval alone | 49% |
| Contextual retrieval + reranking | 67% |
Syntax
chunks = split_into_chunks(document)
contextualized = [prepend_context(chunk, document) for chunk in chunks]
embeddings = embed(contextualized)
vector_store.upsert(embeddings)Examples
# Before: a chunk that's meaningless once isolated from its document
"The company's revenue grew by 3% over the previous quarter."
# After contextual retrieval: the chunk carries its own context,
# so its embedding actually encodes what it's about.
"This chunk is from an SEC filing on ACME Corp's performance in
Q2 2023; the company's revenue grew by 3% over the previous quarter."The retrieval failure is often a chunking problem, not an embedding-model problem
Before switching embedding models or tuning similarity thresholds to fix poor retrieval, check whether chunks carry enough standalone context to be found at all. Anthropic's own measured improvement came from fixing the chunk's context, not the embedding model.
Visual Diagram
Common Mistakes
- Chunking a document by a fixed size (characters or tokens) without regard to whether each resulting chunk still makes sense in isolation.
- Assuming a retrieval failure means the embedding model is wrong, when the actual cause is chunks that lost essential context (which document, which section) during chunking.
- Stuffing an entire knowledge base into the prompt "to be safe," instead of building real retrieval, wasting context window and degrading accuracy on irrelevant content.
- Skipping reranking entirely; contextual retrieval alone helps substantially, but reranking on top of it produced a materially larger measured improvement.
Performance
- Contextualizing every chunk before embedding adds real one-time preprocessing cost at ingestion, paid once per document, not on every query.
- Retrieval-time cost scales with the size of the vector index searched, not the size of the original knowledge base, which is the entire point of pre-processing into chunks and embeddings ahead of time.
Best Practices
- Prepend chunk-specific explanatory context before embedding, so each chunk is self-contained rather than dependent on its original surrounding text to be understood.
- Add a reranking pass over retrieved candidates; combined with contextual chunking, it produced a substantially larger measured improvement than either alone.
- Treat a retrieval failure as a chunking/context problem to investigate first, before assuming the embedding model or similarity threshold is at fault.
- Size chunks and retrieved-context volume deliberately against the model's context window, retrieval should reduce what needs to fit, not just relocate the same volume problem.