Skip to content
anythingai.space
Module 07

Retrieval & RAG

Chunking, embeddings, vector databases, hybrid search, reranking, GraphRAG, agentic RAG and RAG at scale.

lessons
37
steps
332
time
444 min
OFFLINE · INDEXINGONE DOC → CHUNKS§1 Eligibility§2 Refund window: 14 days§3 Annual plans§4 ExceptionsPolicy docs1,200 filesChunker~400 tokensEmbedderVector DB48k chunks
01beginner·12 min·9 steps

RAG Fundamentals

Give a model the right facts at question time instead of hoping it memorised them — and know when not to.

  • Why a model alone can't answer questions about your data
  • The two halves of RAG: offline indexing and online retrieval
  • Where RAG fails: the three retrieval gaps
  • When a long, cached context beats retrieval
Start →
100-TOKEN CHUNKS · 60 SHARP VECTORSc1c2c3c4c5c6c7c8c9c10…c60precise match · but “It costs $200” — whatcosts $200?1,000-TOKEN CHUNKS · 6 DILUTED VECTORSchunk 1chunk 2 · mixedchunk 3chunk 4chunk 5chunk 6full context · but the price is one sentencein a thousand tokens
02beginner·11 min·8 steps

Chunking Strategies

Where you cut a document decides what retrieval can ever find — split on meaning and structure, not on a character count.

  • Why fixed-size chunks break meaning, even with overlap
  • The size tension: small chunks find, large chunks think
  • Recursive and semantic splitting on real boundaries
  • Parent-child chunking: index small, return large
Start →
ONE SHARED EMBEDDING SPACEdoc vectorsquery vectorCorpusindexed oncevoyage-4-large$0.12 / 1M · offlineQueryhot pathvoyage-4-lite$0.02 / 1M · onlineVector index
03intermediate·12 min·9 steps

Embedding Models

How text becomes a vector, why you rarely need all of its dimensions, and how to pick and upgrade the model behind your index.

  • What an embedding model does and why nearby means similar
  • Matryoshka embeddings: search with a prefix, refine with the full vector
  • Binary and int8 quantization and what they save
  • How to read leaderboards and pick a model
Start →
LAYER 2 · SPARSELAYER 1LAYER 0 · ALL VECTORSlong jumpdescenddescendv1entryv9v1v4v9v1v3v4v7nearestv9
04intermediate·13 min·9 steps

Vector Databases

How a vector database finds nearest neighbours among millions of embeddings in milliseconds — and what that costs in RAM, recall and operations.

  • Why brute-force search stops scaling and what ANN trades away
  • How HNSW's layered graph finds neighbours
  • The RAM wall, and how DiskANN and low-bit quantization get past it
  • Why metadata filtering is the hidden bottleneck
Start →
querytop-1HOW THE IDENTIFIER GETS TOKENIZEDConfigureNVIDIA_VISIBLE_DEVICESFragments with little meaning of their own →a vague vectorUserDense searchembeddings · meaningmissed“GPU drivers”close in meaning
05intermediate·11 min·9 steps

Hybrid Search

Run keyword and vector search side by side and fuse their rankings, so exact identifiers and paraphrases both make the top-k.

  • Which queries dense search misses and BM25 catches (and vice versa)
  • How Reciprocal Rank Fusion merges two rankings
  • Why adding raw scores is dangerous, and how alpha weighting works
  • Learned sparse vectors (SPLADE) as the keyword arm
Start →
by relevancediverse top 5RERANKER OUTPUTguide.md §2guide.md §2 (copy)guide.md §2 (v1)FAQ: refundspolicy 2026AFTER MMR + FRESHNESS BOOSTpolicy 2026boostedguide.md §2FAQ: refundsCross-encoderMMR + boostdiversity · freshnessLLM
06intermediate·12 min·9 steps

Reranking Strategies

Retrieve wide and cheap, then let a cross-encoder read each candidate with the query and put the right chunk first.

  • Why a bi-encoder ranks close candidates in the wrong order
  • How a cross-encoder scores query and document together
  • Sizing the funnel: how many candidates to rerank within a latency budget
  • Reading calibrated scores: a threshold instead of a fixed top-k
Start →
COMMUNITY · PEOPLECOMMUNITY · DELIVERYCOMMUNITY · FINANCE“Platform team staffsAlpha”“Alpha relies on VendorX”“Q3 cuts hit projectlines”Engineer ApersonPlatform teamorg unitProject AlphaprojectVendor XsupplierQ3 budgetfunding lineCFO memocuts Q3 budget
07advanced·12 min·9 steps

GraphRAG

When the answer lives in the connections between documents, extract entities into a graph — but only after your failures prove you need one.

  • Which questions vector RAG structurally cannot answer
  • The 30% rule: decide from tagged failures, not aesthetics
  • Extract → build → query: local search over entities and edges
  • Community summaries for global questions
Start →
hop 1, hop 2enoughAGENT STATEhop 1: Microsofthop 2: RedmondUserAgentplans the next searchSearch toolhybrid + rerank2 hopsEnough?sufficiency checkAnswer
08advanced·12 min·9 steps

Agentic RAG

Turn the fixed retrieve-then-generate pipeline into a loop that decides what to search next and when it has enough — with a budget.

  • Why one retrieval can't answer multi-hop questions
  • The agent loop: plan, retrieve, check, repeat
  • Self-RAG and Corrective RAG as built-in quality checks
  • Adaptive RAG: a cheap path first, escalate on a measured signal
Start →
RAW CHUNKIt costs $200.what does?CONTEXTUALIZED CHUNKThis chunk is about pricingfor Widget-Xin North America.It costs $200.FEWER TOP-20 RETRIEVAL FAILURES (ANTHROPIC)+ contextual embeddings35%+ contextual BM2549%+ reranker67%
09intermediate·11 min·9 steps

Advanced Retrieval Patterns

Close the gap between short questions and long documents: decompose queries, embed hypothetical answers, enrich chunks, and know what each trick costs.

  • The asymmetry problem: short queries vs long chunks
  • HyDE — and why to fuse it with the real query
  • Decompose compound questions into sub-queries
  • Enrich chunks with summaries and generated questions
Start →
top-20SOURCE DOCUMENTAcme Corp Q3 2025 Financial Report§4 Product PricingThe Standard plan…CHUNK 17, AS INDEXEDIt costs $200/month.what costs $200?User“Acme Standard plan cost?”Vector indexmissOther chunksshare the query words
10intermediate·11 min·9 steps

Contextual Retrieval

Before indexing a chunk, prepend a short note on where it came from — and cut retrieval failures by up to two-thirds.

  • Context dilution: why a chunk can't be found once it's cut out
  • How an LLM-written context prefix fixes both dense and BM25 retrieval
  • The 35% / 49% / 67% failure reductions, stage by stage
  • Keeping ingestion cheap with prompt caching and small models
Start →
QUERY → TOKEN VECTORS (ONLINE)what128-dis128-dthe128-dprice128-dof128-dWidget-X128-dDOCUMENT → TOKEN VECTORS (OFFLINE, STORED)Widget-X128-dcosts128-d$200128-dper128-dmonth128-dBi-encodernever interactLate interactiontoken-level MaxSimCross-encoderfull joint attention
11advanced·12 min·8 steps

Late Interaction & ColBERT

Keep one small vector per token and match token to token at query time — near cross-encoder accuracy with documents still encoded offline.

  • Where late interaction sits between bi-encoders and cross-encoders
  • How MaxSim scores a query against per-token document vectors
  • Why storage explodes, and how ColBERTv2 compresses it
  • PLAID and MUVERA: making multi-vector search fast
Start →
QueryText indexhybridTable indexMarkdown + metadataImage indexpage / figure vectorsRRF + rerankVLM
12advanced·12 min·9 steps

Multimodal RAG

Retrieve charts, tables, page layouts and video — not just the text a parser managed to extract.

  • What text-only RAG throws away on real documents
  • The three architectures: unified space, per-modality fusion, page-as-image
  • Why ColPali's index is large, and when one vector per page is enough
  • Dual representations for charts and atomic tables
Start →
OFFLINE · TEST SETONLINE · LIVE TRAFFICbatchper-step spansTHREE LAYERS OF EVIDENCE — FAST AND NARROW → SLOW AND MEANINGFULcomponent scoresstep tracesresolution · CSAT · escalationTest setQ · context · answerRAGASLLM-judged scorersRAG pipelineTruLenstrace + feedback
13intermediate·13 min·10 steps

RAG Evaluation Patterns

Turn “the answers are sometimes wrong” into a measurable root cause with the RAG triad, component metrics, calibrated judges and regression gates.

  • The RAG triad: three checks between query, context and answer
  • Diagnosing a failure from its triad pattern
  • Using LLM judges without trusting them blindly
  • RAGAS, TruLens and the business metrics above them
Start →
WHERE A TYPICAL QUERY'S MONEY GOESLLM generation ~85%reranksearch ~2% · embed ~1%GENERATION COST PER REQUEST (2.6K IN / 300 OUT)Small · GPT-6 Luna~$0.0004Mid · Sonnet 5.5~$0.008 (20×)Large · Opus 5.5~$0.016 (2× mid)
14advanced·14 min·9 steps

Production RAG at Scale

Make RAG fast, cheap and safe for many tenants: parallel retrieval, routing, three cache layers, model tiering, isolation and the alerts that catch drift.

  • Cut latency by running retrieval arms in parallel
  • Route queries and cache at three layers
  • Where the money goes — and model tiering
  • Tenant isolation, including the leaks a filter doesn't stop
Start →
WHAT EACH TIER REMOVESdoc Akeptdoc A (copy)exactdoc A (re-templated)MinHashdoc A (paraphrased)SemDeDupLexical dedup can wrongly drop documentsthat share a template but differ in meaningRecordsExact hashMinHash + LSHSemDeDupembeddings
15intermediate·12 min·9 steps

Data Engineering for AI

One pipeline feeds both RAG and fine-tuning: detect, parse, clean, dedup, govern and filter — then keep it fresh and auditable.

  • The shared trunk: one pipeline, two consumers
  • Route pages to the cheapest parser tier that works
  • The dedup cascade and its three-way payoff
  • Governance as metadata on every record
Start →
TRAFFIC SPLIT AFTER TIERING (ILLUSTRATIVE)routine → small / open modelhard → frontierGolden setyour questions + chunksHosted frontierOpen modelScorecardgrounded · p95 · $/answer
16beginner·11 min·9 steps

Choosing an LLM for RAG

The generator is one swappable box in a RAG pipeline — pick it by hosting, context size, modality, language and cost, then prove the choice on your own questions.

  • What the LLM does in a RAG pipeline, and the knobs that matter
  • Hosted APIs vs open weights you run yourself
  • Matching model strengths (long context, multimodal, code, languages) to the job
  • Choosing with an eval set and routing between tiers
Start →
chunksWithout a framework: glue code,retries and client libraries foreach boxLoaderPDF · web · DBSplitterchunksVector storeRetrieverPrompt templateLLM
17beginner·11 min·9 steps

RAG Frameworks & Model Access

LangChain, LlamaIndex and Haystack package the plumbing of a RAG pipeline; Ollama, Hugging Face and OpenRouter decide where the models come from.

  • The building blocks every RAG framework provides
  • How LangChain, LlamaIndex and Haystack differ in their core abstraction
  • Running models locally with Ollama and Hugging Face
  • Routing across providers with a unified API and fallbacks
Start →
PARSING SERVICE (CLOUD)uploadTABLE SURVIVES AS MARKDOWN| Region | 2025 || APAC | 12.4 || EMEA | 9.8 |Annual report.pdftables · charts · 2 columnsLlamaParsetables · images · layoutClean Markdownheadings · tables kept
18beginner·12 min·10 steps

Document Parsing & Extraction Tools

Turn PDFs, scans and websites into clean, structured text a retriever can use — with layout parsers, cloud document services, LLM extraction and LLM-aware crawlers.

  • Why plain text extraction breaks tables and layout
  • Layout-aware parsers: LlamaParse, Docling and MegaParser
  • Managed processors and LLM-based extraction into a schema
  • Getting clean content from websites with Firecrawl and ScrapeGraph AI
Start →
CHUNK 1…policy §4.Refunds are allowedwithin 14 daysCHUNK 2— except annual plans,which require written approval.§5 Invoices…✂ cut lands mid-rule: the exception is orphaned
19intermediate·12 min·10 steps

Chunking Mechanics

The details that decide where a chunk starts and stops: sliding windows and overlap, token-accurate sizing, similarity merging, and keeping pages, lists and figures intact.

  • Sliding windows and how much overlap to use
  • Sizing chunks in tokens with the model's own tokenizer
  • Merging sentence micro-chunks by similarity
  • Keeping sections, pages, lists and figures as whole units
Start →
“Overview?” → summary“Notice period?” →paragraphSummary chunkwhole leave policy§ Annual leave§ Parental leave¶ carry-over¶ notice period¶ eligibility
20intermediate·12 min·9 steps

Adaptive Chunking & Tuning

Stop guessing one chunk size for everything: adapt chunking per domain and per query, keep several granularities, and tune with offline sweeps, chunk-quality scores and live A/B tests.

  • Per-domain chunking presets chosen at ingestion
  • Multi-scale and query-aware chunking at retrieval time
  • Finding the right size with offline sweeps and chunk-quality scores
  • Proving a change with live A/B tests and a continuous feedback loop
Start →
COGNITA · MODULAR PLATFORMLLMWARE · ONE LIBRARYLoaders + parsersEmbedderpluggableVector storeParse documentsSmall embedderruns privatelyRetrieve
21beginner·11 min·10 steps

Embedding Providers

OpenAI, Gemini, Cohere, Jina, Nomic, Ollama and RAG platforms all turn text into vectors — they differ in hosting, languages, modalities and how they treat queries versus documents.

  • Why vectors from different providers can never be mixed
  • Hosted embedding APIs and their query/document modes
  • Open and local embedders: Nomic and Ollama
  • Choosing a provider — and what switching costs
Start →
“REFUND POLICY” → 2 TOKENSrefund▁policy“ERR_CONN_0X8004” → 8 TOKENSERR_CONN_0x8004SAME SENTENCE IN HINDI → ~3× THE TOKENS OF ENGLISHरिफंड…Over max length? The tail is cut off beforeembedding — silently
22advanced·13 min·9 steps

Embedding Optimization

Make the embedder understand your corpus: respect tokenization, pool token vectors the right way, mix granularities, and adapt the model with contrastive fine-tuning on in-domain data.

  • The levers for improving an embedder without changing the rest of RAG
  • How tokenization and pooling shape the final vector
  • Combining sentence, paragraph and document vectors
  • Contrastive fine-tuning with hard negatives on in-domain data
Start →
RAG appProxyQuery nodesscale for QPS×4Data nodesscale for ingest×2Index nodesbuild HNSW / IVF×2Object storage
23beginner·12 min·10 steps

The Vector Store Landscape

From an in-process library to a fully managed service: what FAISS, Chroma, Postgres with pgvector, Milvus, Weaviate and Pinecone each take off your hands, and when to pick which.

  • What a database has to do for RAG, beyond nearest-neighbour search
  • Libraries and embedded stores: FAISS and Chroma
  • Adding vectors to Postgres with pgvector
  • Dedicated and managed databases: Milvus, Weaviate, Pinecone
Start →
ZONE AZONE BQuery routerscatter · gatherShard 1Shard 2Shard 3Replica 1Replica 2Replica 3
24intermediate·12 min·8 steps

RAG Storage Design

Design the storage layer behind retrieval: a metadata schema that powers filters and access control, versioned documents, incremental refresh, real deletion, and sharding with replication.

  • The decisions that make up a RAG storage layer
  • Designing metadata that serves filtering, routing and access control
  • Versioning documents and refreshing the index incrementally
  • Deleting data for real, and scaling with shards and replicas
Start →
CONVERSATION MEMORYproduct: analytics add-onplans discussed: Team → Businesstopic: pricingTURN 3 AS TYPEDhow much isitforthe bigger one?RESOLVEDhow much isthe analytics add-ononthe Business plan?
25intermediate·14 min·10 steps

Query Understanding

Users type vague, conversational, multi-part questions. Before retrieval: skip RAG when it isn't needed, resolve references, clarify ambiguity, rewrite, classify, route, decompose — and normalise the query vector.

  • Why raw user queries make poor search queries
  • Skipping RAG for small talk and resolving conversational references
  • Disambiguating and rewriting into a standalone query
  • Classifying, routing and decomposing queries
Start →
ORIGINALheart attack follow-upEXPANDED FORMSmyocardial infarction follow-upglossaryMI aftercareglossarycare plan after a heart attackparaphraseEach form is searched; hits are merged
26intermediate·12 min·9 steps

RAG Architecture Variants

Beyond one query and one retriever: expand the query, run many variants, fuse many retrievers, search a document hierarchy top-down, and refine candidates through a multi-stage funnel.

  • Expanding one query into many to raise recall
  • Multi-query RAG and RAG Fusion: many queries, many retrievers, one ranked list
  • Hierarchical RAG for long, structured documents
  • The multi-stage funnel from broad candidates to final context
Start →
DOCUMENT AGE VS RECENCY MULTIPLIER (HALF-LIFE 1 YEAR)Handbook 2026×1.0Wiki page 2024×0.78Policy 2021filteredyears old →CandidatesTemporal filteris_latest · as-of dateRecency decayhalf-life per type
27intermediate·12 min·9 steps

Ranking Beyond Relevance

Relevance scores aren't the whole ranking. Add domain rules, recency, field weights, cluster diversity and safety labels — and measure every change with recall, precision, MRR and nDCG.

  • Encoding authority and document type as scoring rules
  • Recency decay and temporal filters
  • Weighting fields and diversifying by cluster
  • Demoting or excluding unsafe content
Start →
8K-TOKEN WINDOWsystem · 12%history (last 10 turns) · 25%top 6 chunks · 45%query · 3%answer room · 15%Over budget? Drop the oldest turns, then thelowest-ranked chunksNever trim the system instructions or thequery
28intermediate·12 min·9 steps

Context Assembly

Retrieval hands you a pile of chunks; context assembly turns it into a prompt the model can use — budgeted, de-duplicated, merged, ordered and laid out in labelled sections.

  • Why a flat list of chunks makes a poor prompt
  • Splitting the context window between instructions, history and knowledge
  • Removing duplicates and merging fragments into coherent passages
  • Ordering evidence and laying it out in labelled sections
Start →
ONE PROMPT STRINGrulesretrieved text (incl. “ignore the above…”)40 turns of chatQwindowRules and evidence look the sameHistory crowds out evidence
29intermediate·11 min·8 steps

Prompt Layout & Conversation History

Put retrieved evidence in the right message role, tell the model how to stay grounded and cite, ask for a fixed answer structure, and keep long conversations inside the window by truncating and selecting history.

  • Separating instructions, evidence and the question by message role
  • Writing grounded-answer and citation instructions
  • Asking for a consistent answer structure
  • Keeping long chats in budget with truncation and query-guided history
Start →
MODULE CHUNK · PAYMENT-SERVICE.TScallsRepositoryservices · testsimportsgateway · retry libclass PaymentServicecharge()retryCharge()
30intermediate·11 min·8 steps

Code, Tables & Images in Context

Code, tables and page images break prose-style RAG. Chunk code by module, file and diff; then hand the model fenced code, trimmed tables and page images with their text.

  • Chunking code by class, module and file — with repository metadata
  • Using diff hunks as chunks for change-focused questions
  • Assembling code and tables so the model can read them
  • Pairing page images with their text in multimodal context
Start →
TYPICAL FAILURERefunds take 30 daysnot in context(no citation)free-form prose{"answer": … truncatedAssembled promptrules + [1]–[5] + questionGeneratorAnswerno cites
31intermediate·12 min·9 steps

Answer Generation

The last step of RAG: generate an answer that stays inside the evidence, cites it claim by claim, follows a template or JSON schema, reasons privately, and synthesises across documents.

  • The contract a RAG answer must meet
  • Grounded, citation-based generation
  • Templates, JSON mode and schema enforcement
  • Reasoning privately and synthesising across documents
Start →
DRAFT ANSWER, CLAIM BY CLAIM14-day window [1]annual plans need approval [1]refund arrives in 3 days [2][2] says nothing about timingContext[1]–[5]GeneratorDraft answer1 unsupportedUser
32advanced·13 min·10 steps

Verified Answers

Grounded prompts reduce hallucinations; checks catch the rest. Detect unsupported claims, run a verification pass, score citations, express uncertainty, and enforce compliance and harm rules before an answer ships.

  • Detecting hallucinations against the retrieved context
  • Two-pass generation and citation quality checks
  • Layered mitigation and honest uncertainty
  • Compliance and harm rules on the way out
Start →
PRESSURE → PATTERNcost → cascadelatency → streamingcomplexity → orchestrationfailures → retry + fallbackdebugging → telemetryUserUnderstandRetrieveRankGenerate
33intermediate·12 min·7 steps

RAG Pipeline Patterns

How production RAG pipelines are wired: cascades that escalate only when needed, streaming that answers while retrieval finishes, orchestration graphs, retries and fallbacks, and telemetry on every stage.

  • The stages of a RAG pipeline and the shapes they can take
  • Cascades: cheap first, escalate on low confidence
  • Streaming answers before retrieval fully finishes
  • Orchestration graphs, retries and fallbacks
Start →
EU REGIONUS REGIONEU tenantUS tenantRegion routerby tenant residencyEU vector storeEU LLM endpointUS vector storeUS LLM endpoint
34intermediate·13 min·9 steps

Access Control & Compliance

A RAG system will happily retrieve anything it indexed. Govern it with layered controls: roles and attributes, permission-aware retrieval, tenant isolation, data residency and the industry frameworks that demand them.

  • Why RAG needs governance, and the layers it is built from
  • Role-based and attribute-based access rules
  • Enforcing permissions inside retrieval, not after it
  • Tenant isolation and data residency
Start →
RAW TICKET TEXTCustomerC-2231emailedc.doe@example.comcard4111 1111 1111 1111AFTER MASKINGCustomer[CUSTOMER_ID]emailedc***@example.comcard[REDACTED]Runs before indexing, before the LLM calland before logging
35advanced·13 min·8 steps

RAG Safety & Security

Protect a RAG system where it is attacked and where it leaks: mask PII before it is stored or sent, keep it out of embeddings, filter inputs and outputs, defend the knowledge base against poisoning, and red-team it all.

  • Where a RAG system can be attacked or leak data
  • Detecting and masking PII, and keeping it out of embeddings
  • Filtering inputs and defending the knowledge base against poisoning
  • Checking outputs and red-teaming the whole pipeline
Start →
NO VISIBILITYQUESTIONS YOU CAN'T ANSWERrewritten query?which chunks?what scores?which prompt version?where did 4 s go?UserRewriteRetrieveRerankGenerate
36intermediate·11 min·8 steps

Tracing & Logging for RAG

See inside every RAG request: metrics, logs and traces; structured, PII-safe log records; and one distributed trace per query with a span for each pipeline stage.

  • Metrics, logs and traces — and what each tells you about a RAG system
  • Structured log records you can filter and join
  • Keeping personal data out of logs
  • Tracing one query across services with a span per stage
Start →
DRILL DOWN: ERRORS BY INDEX VERSIONkb-v11: 0.3%kb-v12: 7.8%→ deployed 09:58Latency p95TrafficErrorsspikeSaturationCost / requestHit-rateHelpful rate
37intermediate·12 min·10 steps

RAG Metrics, Alerts & Feedback

Know whether a RAG system is healthy and helpful: core and quality metrics, explicit and implicit user signals, retrieval hit-rate analytics, a segmented dashboard, SLO alerts and quality-regression alerts.

  • The core system metrics and the quality metrics above them
  • Explicit feedback and implicit behavioural signals
  • Retrieval hit-rate and per-source analytics
  • Dashboards, SLO alerts and quality-regression alerts
Start →
Read deeper

Guide chapters · Retrieval Systems

Next module · 08
Agentic Systems