Skip to content
anythingai.space
Retrieval & RAGbeginner11 min · 9 steps

Choosing an LLM for RAG

The generator is one swappable box in a RAG pipeline — pick it by hosting, context size, modality, language and cost, then prove the choice on your own questions.

questionchunksWHAT TO CHECK ON EVERY CANDIDATE MODELcontext windowoutput limittemperature ≈ 0–0.3API or localprice per tokenlanguagesmodalitiesUserRetrievertop-k chunksPromptinstructions + chunksGenerator LLM?
requestdatatokenresponse
1 / 9
Key ideaStep 1 of 9

The generator is one swappable box

Retrieval finds the evidence; the LLM turns question + retrieved chunks into an answer. It is the last stage of the pipeline and usually the most expensive one.

Four knobs decide whether a model fits: the context window (how many chunks fit), token limits on output, temperature (keep it low for factual answers) and access — a cloud API or a model you run on your own hardware.

InsightKeep the generator behind one interface. You will swap it more often than any other component.
What changed
  • +Generator slot in the pipeline

Lesson 16 of 37 in Retrieval & RAG