RAG Pipeline Cheat Sheet

Chunking, embedding, retrieval and reranking — the decisions that matter

Retrieval-augmented generation in one page: the stages, the parameters worth tuning, sensible starting values, and the failure modes that make a RAG system look broken when it is only mis-tuned.

Category: RAG Difficulty: Advanced Version: 1.0 Updated: July 31, 2026 Author: Sabir

The pipeline

Every RAG system is these six stages. Most quality problems are in stages 2 and 5.

Stage	What happens	Where it goes wrong
1. Ingest	Load and clean source documents	PDF extraction mangles tables and columns
2. Chunk	Split into retrievable units	Splitting mid-idea destroys meaning
3. Embed	Turn chunks into vectors	Mixing embedding models between index and query
4. Store	Index vectors plus metadata	No metadata means no filtering later
5. Retrieve	Find candidates for the question	Top-k too low, or pure vector search with no keyword fallback
6. Generate	Answer using retrieved context	No instruction to say "I do not know"

Starting values

Defaults that work for prose documents. Tune from here rather than from zero.

Parameter	Start with	Notes
`chunk_size`	600–800 tokens	Smaller for dense reference, larger for narrative
`chunk_overlap`	10–15% of chunk_size	Stops an idea being cut in half at a boundary
`top_k`	20 retrieved → 5 after rerank	Retrieve wide, then narrow — not the other way around
`hybrid_alpha`	0.5	Balance of vector vs keyword; raise for jargon-heavy corpora
`min_score`	unset at first	Add only once you have measured a real threshold

Failure modes

Confidently wrong answers — Add "answer only from the context; if it is not there, say so" and show sources.
Right document, wrong passage — Your chunks are too large. Halve chunk_size and add a reranker.
Never finds obvious matches — Pure vector search misses exact terms. Add keyword/BM25 hybrid retrieval.
Good in testing, poor in production — Test questions matched your chunk boundaries. Test with real user phrasing.
Slow responses — Rerank fewer candidates, cache embeddings, and stream the answer.

Code examples

Chunking with overlap Python

Sentence-aware splitting beats fixed character windows.

def chunk(text, size=700, overlap=100):
    """Split on sentence ends so an idea is never cut in half."""
    import re
    sentences = re.split(r'(?<=[.!?])\s+', text.strip())
    chunks, current = [], ''

    for s in sentences:
        if len(current) + len(s) > size and current:
            chunks.append(current.strip())
            # carry the tail forward so context survives the boundary
            current = current[-overlap:] + ' '
        current += s + ' '

    if current.strip():
        chunks.append(current.strip())
    return chunks

FAQs

Do I need a vector database?

Not below roughly 10,000 chunks — an in-memory index or a Postgres extension is simpler and fast enough.

Is a bigger context window a replacement for RAG?

No. Cost scales with what you send, and retrieval quality still beats dumping everything into the prompt.

Resources

AI Workflows — /workflows