Retrieval-Augmented Generation (RAG)

RAG is a technique that enhances LLM responses by retrieving relevant documents from an external knowledge base and injecting them into the prompt context before generation.

What It Really Means

Large language models are trained on static datasets with a knowledge cutoff date. They cannot access your company's internal documentation, latest product specs, or real-time data. RAG solves this by adding a retrieval step before generation: instead of relying solely on the model's parametric memory, you search a knowledge base for relevant context and include it in the prompt.

The core insight is separation of concerns. The retrieval system handles what to know (finding relevant documents), while the LLM handles how to respond (synthesizing information into a coherent answer). This makes the system modular — you can update your knowledge base without retraining the model, swap out the retrieval engine without changing the LLM, or upgrade the LLM without rebuilding your index.

RAG emerged from a 2020 paper by Lewis et al. at Facebook AI Research. The original architecture used a DPR (Dense Passage Retriever) combined with a BART generator. Modern RAG systems have evolved significantly, using vector embeddings for retrieval, rerankers for precision, and sophisticated chunking strategies for document processing.

How It Works in Practice

A RAG pipeline has two phases: indexing (offline) and retrieval + generation (online).

Indexing Phase

1. Load documents — PDFs, HTML pages, Markdown files, database records
2. Chunk documents — Split into smaller segments (typically 256-1024 tokens). See chunking strategies for approaches.
3. Embed chunks — Convert each chunk into a vector using an embedding model
4. Store in vector database — Index vectors for fast similarity search via semantic search

Query Phase

1. Embed the user query — Same embedding model as indexing
2. Retrieve relevant chunks — Find top-k nearest neighbors in vector space
3. Rerank (optional) — Use a cross-encoder to reorder results by relevance
4. Augment the prompt — Insert retrieved chunks into the LLM prompt
5. Generate response — LLM synthesizes an answer grounded in the retrieved context

Concrete Example: Internal Documentation Bot

A developer asks: "How do I configure SSO for our staging environment?"

1. Query gets embedded to a 1536-dimensional vector
2. Vector search finds 5 chunks from internal docs about SSO configuration
3. Reranker filters down to the 3 most relevant chunks
4. Prompt becomes: "Based on the following documentation, answer the user's question. [chunk1] [chunk2] [chunk3] Question: How do I configure SSO for our staging environment?"
5. LLM generates a step-by-step answer citing the documentation

Implementation

Trade-offs

When to Use RAG

• Knowledge changes frequently (docs, product info, policies)
• You need source attribution and citations
• Domain-specific data that the base model has never seen
• You need to control and update the knowledge base without retraining
• Budget constraints prevent fine-tuning

When NOT to Use RAG

• The task requires reasoning patterns, not knowledge retrieval (use fine-tuning)
• Latency requirements are very tight (retrieval adds 100-500ms)
• The answer requires synthesizing information across dozens of documents
• The knowledge is already well-represented in the base model's training data

Advantages

• No model training required — faster to deploy
• Knowledge stays fresh by updating the index
• Reduces hallucination by grounding responses in source documents
• Source attribution is straightforward

Disadvantages

• Retrieval quality is a bottleneck — garbage in, garbage out
• Adds latency compared to direct LLM calls
• Chunk boundaries can split relevant context
• Requires maintaining a vector database and embedding pipeline
• Context window limits cap how much retrieved content you can include (see token budgeting)

Common Misconceptions

• "RAG eliminates hallucination" — RAG reduces hallucination but does not eliminate it. The LLM can still misinterpret retrieved context, confuse details across chunks, or generate plausible-sounding answers that contradict the source material. You still need AI guardrails.

• "More retrieved chunks = better answers" — Stuffing the context window with 20 chunks often degrades performance. The LLM may get confused by contradictory or marginally relevant information. Empirically, 3-5 high-quality chunks outperform 15 mediocre ones.

• "RAG and fine-tuning are mutually exclusive" — They are complementary. You can fine-tune a model to better follow your RAG prompt format while using RAG for dynamic knowledge. See fine-tuning vs RAG for a detailed comparison.

• "Embedding similarity guarantees relevance" — Cosine similarity in embedding space is a heuristic. Two chunks can be semantically similar but not relevant to the specific question. This is why reranking is critical.

• "You only need a vector database" — Production RAG systems often combine vector search with keyword search (hybrid retrieval), metadata filtering, and reranking for best results.

How This Appears in Interviews

RAG is one of the most common topics in AI engineering interviews. Expect questions like:

• "Design a RAG system for internal documentation" — focus on chunking strategy, embedding model selection, retrieval pipeline, and how you handle updates. Check our interview questions on RAG for practice.
• "How would you evaluate RAG quality?" — discuss retrieval metrics (recall@k, MRR) vs generation metrics (faithfulness, relevance). See our guide on AI engineering.
• "Your RAG system returns irrelevant results. How do you debug?" — walk through the retrieval pipeline: check embedding quality, chunk boundaries, query reformulation, and reranking.
• "When would you choose RAG over fine-tuning?" — demonstrate understanding of the fine-tuning vs RAG trade-offs.

Related Concepts

• Vector Embeddings — The mathematical foundation of RAG retrieval
• Chunking Strategies for RAG — How to split documents for optimal retrieval
• Semantic Search — The retrieval mechanism powering RAG
• Fine-Tuning vs RAG — When to choose each approach
• Hallucination in LLMs — The problem RAG helps mitigate
• Embedding Models — Choosing the right model for your RAG pipeline
• Algoroq Pricing — Start practicing RAG system design questions