Retrieval & RAG
Chunking, embeddings, vector databases, hybrid search, reranking, GraphRAG, agentic RAG and RAG at scale.
- lessons
- 37
- steps
- 332
- time
- 444 min
RAG Fundamentals
Give a model the right facts at question time instead of hoping it memorised them — and know when not to.
- Why a model alone can't answer questions about your data
- The two halves of RAG: offline indexing and online retrieval
- Where RAG fails: the three retrieval gaps
- When a long, cached context beats retrieval
Chunking Strategies
Where you cut a document decides what retrieval can ever find — split on meaning and structure, not on a character count.
- Why fixed-size chunks break meaning, even with overlap
- The size tension: small chunks find, large chunks think
- Recursive and semantic splitting on real boundaries
- Parent-child chunking: index small, return large
Embedding Models
How text becomes a vector, why you rarely need all of its dimensions, and how to pick and upgrade the model behind your index.
- What an embedding model does and why nearby means similar
- Matryoshka embeddings: search with a prefix, refine with the full vector
- Binary and int8 quantization and what they save
- How to read leaderboards and pick a model
Vector Databases
How a vector database finds nearest neighbours among millions of embeddings in milliseconds — and what that costs in RAM, recall and operations.
- Why brute-force search stops scaling and what ANN trades away
- How HNSW's layered graph finds neighbours
- The RAM wall, and how DiskANN and low-bit quantization get past it
- Why metadata filtering is the hidden bottleneck
Hybrid Search
Run keyword and vector search side by side and fuse their rankings, so exact identifiers and paraphrases both make the top-k.
- Which queries dense search misses and BM25 catches (and vice versa)
- How Reciprocal Rank Fusion merges two rankings
- Why adding raw scores is dangerous, and how alpha weighting works
- Learned sparse vectors (SPLADE) as the keyword arm
Reranking Strategies
Retrieve wide and cheap, then let a cross-encoder read each candidate with the query and put the right chunk first.
- Why a bi-encoder ranks close candidates in the wrong order
- How a cross-encoder scores query and document together
- Sizing the funnel: how many candidates to rerank within a latency budget
- Reading calibrated scores: a threshold instead of a fixed top-k
GraphRAG
When the answer lives in the connections between documents, extract entities into a graph — but only after your failures prove you need one.
- Which questions vector RAG structurally cannot answer
- The 30% rule: decide from tagged failures, not aesthetics
- Extract → build → query: local search over entities and edges
- Community summaries for global questions
Agentic RAG
Turn the fixed retrieve-then-generate pipeline into a loop that decides what to search next and when it has enough — with a budget.
- Why one retrieval can't answer multi-hop questions
- The agent loop: plan, retrieve, check, repeat
- Self-RAG and Corrective RAG as built-in quality checks
- Adaptive RAG: a cheap path first, escalate on a measured signal
Advanced Retrieval Patterns
Close the gap between short questions and long documents: decompose queries, embed hypothetical answers, enrich chunks, and know what each trick costs.
- The asymmetry problem: short queries vs long chunks
- HyDE — and why to fuse it with the real query
- Decompose compound questions into sub-queries
- Enrich chunks with summaries and generated questions
Contextual Retrieval
Before indexing a chunk, prepend a short note on where it came from — and cut retrieval failures by up to two-thirds.
- Context dilution: why a chunk can't be found once it's cut out
- How an LLM-written context prefix fixes both dense and BM25 retrieval
- The 35% / 49% / 67% failure reductions, stage by stage
- Keeping ingestion cheap with prompt caching and small models
Late Interaction & ColBERT
Keep one small vector per token and match token to token at query time — near cross-encoder accuracy with documents still encoded offline.
- Where late interaction sits between bi-encoders and cross-encoders
- How MaxSim scores a query against per-token document vectors
- Why storage explodes, and how ColBERTv2 compresses it
- PLAID and MUVERA: making multi-vector search fast
Multimodal RAG
Retrieve charts, tables, page layouts and video — not just the text a parser managed to extract.
- What text-only RAG throws away on real documents
- The three architectures: unified space, per-modality fusion, page-as-image
- Why ColPali's index is large, and when one vector per page is enough
- Dual representations for charts and atomic tables
RAG Evaluation Patterns
Turn “the answers are sometimes wrong” into a measurable root cause with the RAG triad, component metrics, calibrated judges and regression gates.
- The RAG triad: three checks between query, context and answer
- Diagnosing a failure from its triad pattern
- Using LLM judges without trusting them blindly
- RAGAS, TruLens and the business metrics above them
Production RAG at Scale
Make RAG fast, cheap and safe for many tenants: parallel retrieval, routing, three cache layers, model tiering, isolation and the alerts that catch drift.
- Cut latency by running retrieval arms in parallel
- Route queries and cache at three layers
- Where the money goes — and model tiering
- Tenant isolation, including the leaks a filter doesn't stop
Data Engineering for AI
One pipeline feeds both RAG and fine-tuning: detect, parse, clean, dedup, govern and filter — then keep it fresh and auditable.
- The shared trunk: one pipeline, two consumers
- Route pages to the cheapest parser tier that works
- The dedup cascade and its three-way payoff
- Governance as metadata on every record
Choosing an LLM for RAG
The generator is one swappable box in a RAG pipeline — pick it by hosting, context size, modality, language and cost, then prove the choice on your own questions.
- What the LLM does in a RAG pipeline, and the knobs that matter
- Hosted APIs vs open weights you run yourself
- Matching model strengths (long context, multimodal, code, languages) to the job
- Choosing with an eval set and routing between tiers
RAG Frameworks & Model Access
LangChain, LlamaIndex and Haystack package the plumbing of a RAG pipeline; Ollama, Hugging Face and OpenRouter decide where the models come from.
- The building blocks every RAG framework provides
- How LangChain, LlamaIndex and Haystack differ in their core abstraction
- Running models locally with Ollama and Hugging Face
- Routing across providers with a unified API and fallbacks
Document Parsing & Extraction Tools
Turn PDFs, scans and websites into clean, structured text a retriever can use — with layout parsers, cloud document services, LLM extraction and LLM-aware crawlers.
- Why plain text extraction breaks tables and layout
- Layout-aware parsers: LlamaParse, Docling and MegaParser
- Managed processors and LLM-based extraction into a schema
- Getting clean content from websites with Firecrawl and ScrapeGraph AI
Chunking Mechanics
The details that decide where a chunk starts and stops: sliding windows and overlap, token-accurate sizing, similarity merging, and keeping pages, lists and figures intact.
- Sliding windows and how much overlap to use
- Sizing chunks in tokens with the model's own tokenizer
- Merging sentence micro-chunks by similarity
- Keeping sections, pages, lists and figures as whole units
Adaptive Chunking & Tuning
Stop guessing one chunk size for everything: adapt chunking per domain and per query, keep several granularities, and tune with offline sweeps, chunk-quality scores and live A/B tests.
- Per-domain chunking presets chosen at ingestion
- Multi-scale and query-aware chunking at retrieval time
- Finding the right size with offline sweeps and chunk-quality scores
- Proving a change with live A/B tests and a continuous feedback loop
Embedding Providers
OpenAI, Gemini, Cohere, Jina, Nomic, Ollama and RAG platforms all turn text into vectors — they differ in hosting, languages, modalities and how they treat queries versus documents.
- Why vectors from different providers can never be mixed
- Hosted embedding APIs and their query/document modes
- Open and local embedders: Nomic and Ollama
- Choosing a provider — and what switching costs
Embedding Optimization
Make the embedder understand your corpus: respect tokenization, pool token vectors the right way, mix granularities, and adapt the model with contrastive fine-tuning on in-domain data.
- The levers for improving an embedder without changing the rest of RAG
- How tokenization and pooling shape the final vector
- Combining sentence, paragraph and document vectors
- Contrastive fine-tuning with hard negatives on in-domain data
The Vector Store Landscape
From an in-process library to a fully managed service: what FAISS, Chroma, Postgres with pgvector, Milvus, Weaviate and Pinecone each take off your hands, and when to pick which.
- What a database has to do for RAG, beyond nearest-neighbour search
- Libraries and embedded stores: FAISS and Chroma
- Adding vectors to Postgres with pgvector
- Dedicated and managed databases: Milvus, Weaviate, Pinecone
RAG Storage Design
Design the storage layer behind retrieval: a metadata schema that powers filters and access control, versioned documents, incremental refresh, real deletion, and sharding with replication.
- The decisions that make up a RAG storage layer
- Designing metadata that serves filtering, routing and access control
- Versioning documents and refreshing the index incrementally
- Deleting data for real, and scaling with shards and replicas
Query Understanding
Users type vague, conversational, multi-part questions. Before retrieval: skip RAG when it isn't needed, resolve references, clarify ambiguity, rewrite, classify, route, decompose — and normalise the query vector.
- Why raw user queries make poor search queries
- Skipping RAG for small talk and resolving conversational references
- Disambiguating and rewriting into a standalone query
- Classifying, routing and decomposing queries
RAG Architecture Variants
Beyond one query and one retriever: expand the query, run many variants, fuse many retrievers, search a document hierarchy top-down, and refine candidates through a multi-stage funnel.
- Expanding one query into many to raise recall
- Multi-query RAG and RAG Fusion: many queries, many retrievers, one ranked list
- Hierarchical RAG for long, structured documents
- The multi-stage funnel from broad candidates to final context
Ranking Beyond Relevance
Relevance scores aren't the whole ranking. Add domain rules, recency, field weights, cluster diversity and safety labels — and measure every change with recall, precision, MRR and nDCG.
- Encoding authority and document type as scoring rules
- Recency decay and temporal filters
- Weighting fields and diversifying by cluster
- Demoting or excluding unsafe content
Context Assembly
Retrieval hands you a pile of chunks; context assembly turns it into a prompt the model can use — budgeted, de-duplicated, merged, ordered and laid out in labelled sections.
- Why a flat list of chunks makes a poor prompt
- Splitting the context window between instructions, history and knowledge
- Removing duplicates and merging fragments into coherent passages
- Ordering evidence and laying it out in labelled sections
Prompt Layout & Conversation History
Put retrieved evidence in the right message role, tell the model how to stay grounded and cite, ask for a fixed answer structure, and keep long conversations inside the window by truncating and selecting history.
- Separating instructions, evidence and the question by message role
- Writing grounded-answer and citation instructions
- Asking for a consistent answer structure
- Keeping long chats in budget with truncation and query-guided history
Code, Tables & Images in Context
Code, tables and page images break prose-style RAG. Chunk code by module, file and diff; then hand the model fenced code, trimmed tables and page images with their text.
- Chunking code by class, module and file — with repository metadata
- Using diff hunks as chunks for change-focused questions
- Assembling code and tables so the model can read them
- Pairing page images with their text in multimodal context
Answer Generation
The last step of RAG: generate an answer that stays inside the evidence, cites it claim by claim, follows a template or JSON schema, reasons privately, and synthesises across documents.
- The contract a RAG answer must meet
- Grounded, citation-based generation
- Templates, JSON mode and schema enforcement
- Reasoning privately and synthesising across documents
Verified Answers
Grounded prompts reduce hallucinations; checks catch the rest. Detect unsupported claims, run a verification pass, score citations, express uncertainty, and enforce compliance and harm rules before an answer ships.
- Detecting hallucinations against the retrieved context
- Two-pass generation and citation quality checks
- Layered mitigation and honest uncertainty
- Compliance and harm rules on the way out
RAG Pipeline Patterns
How production RAG pipelines are wired: cascades that escalate only when needed, streaming that answers while retrieval finishes, orchestration graphs, retries and fallbacks, and telemetry on every stage.
- The stages of a RAG pipeline and the shapes they can take
- Cascades: cheap first, escalate on low confidence
- Streaming answers before retrieval fully finishes
- Orchestration graphs, retries and fallbacks
Access Control & Compliance
A RAG system will happily retrieve anything it indexed. Govern it with layered controls: roles and attributes, permission-aware retrieval, tenant isolation, data residency and the industry frameworks that demand them.
- Why RAG needs governance, and the layers it is built from
- Role-based and attribute-based access rules
- Enforcing permissions inside retrieval, not after it
- Tenant isolation and data residency
RAG Safety & Security
Protect a RAG system where it is attacked and where it leaks: mask PII before it is stored or sent, keep it out of embeddings, filter inputs and outputs, defend the knowledge base against poisoning, and red-team it all.
- Where a RAG system can be attacked or leak data
- Detecting and masking PII, and keeping it out of embeddings
- Filtering inputs and defending the knowledge base against poisoning
- Checking outputs and red-teaming the whole pipeline
Tracing & Logging for RAG
See inside every RAG request: metrics, logs and traces; structured, PII-safe log records; and one distributed trace per query with a span for each pipeline stage.
- Metrics, logs and traces — and what each tells you about a RAG system
- Structured log records you can filter and join
- Keeping personal data out of logs
- Tracing one query across services with a span per stage
RAG Metrics, Alerts & Feedback
Know whether a RAG system is healthy and helpful: core and quality metrics, explicit and implicit user signals, retrieval hit-rate analytics, a segmented dashboard, SLO alerts and quality-regression alerts.
- The core system metrics and the quality metrics above them
- Explicit feedback and implicit behavioural signals
- Retrieval hit-rate and per-source analytics
- Dashboards, SLO alerts and quality-regression alerts
Guide chapters · Retrieval Systems
- RAG Fundamentals8 min
- Chunking Strategies5 min
- Embedding Models9 min
- Vector Databases20 min
- Hybrid Search8 min
- Reranking Strategies11 min
- GraphRAG8 min
- Agentic RAG7 min
- Advanced Retrieval Patterns4 min
- Contextual Retrieval13 min
- Late Interaction & ColBERT11 min
- Multi-Modal RAG14 min
- RAG Evaluation Patterns18 min
- Production RAG at Scale20 min
- Data Engineering for AI14 min