RAG Explained: Retrieval-Augmented Generation

A fluent answer about your company still needs an appropriate source. Retrieval-augmented generation, or RAG, supplies retrieved material at answer time so a model can use it when responding. Retrieval can also find the wrong document, so source checks and evaluation remain part of the work. This guide explains the approach and its limits, within our AI engineering overview.

The problem RAG solves

A model only knows what was in its training data and what you put in its prompt. It cannot know your latest policy, yesterday's incident report or a document written after training finished. And because models are trained to produce fluent text, a gap in knowledge often shows up as a confident wrong answer.

You could paste every relevant document into the prompt, but prompts have limits, and more text is slower, costlier and can distract the model. RAG solves this by finding just the relevant passages first and giving only those to the model.

Where the idea comes from

The name comes from a 2020 paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, by Patrick Lewis and colleagues. They proposed models that combine pre-trained "parametric" memory, meaning what the model learned in training, with "non-parametric" memory, in their case a dense vector index of Wikipedia accessed by a neural retriever. They reported that this produced language that was more specific, diverse and factual than a model that relied on its parameters alone.

Modern systems are built differently, but the principle is the same: look things up, then answer from what you found.

How it works, step by step

Preparing your data (done ahead of time)

1. Ingest your documents: text, PDFs, and with OCR even scanned images.
2. Split them into passages, often called chunks.
3. Index the chunks so they can be searched, commonly by turning each into an embedding, a list of numbers that represents its meaning.

Answering a question (done every time)

1. Search for the passages most relevant to the question.
2. Build a prompt containing the question and those passages, with an instruction to answer only from them.
3. Generate the answer, ideally with references to which passages it used.

The search part is the hard part

The Microsoft AI-103 study guide lists the skills involved: ingesting and indexing documents, images, audio and video; configuring "semantic search, hybrid search, and vector search for grounding"; and enrichment using OCR and layout analysis.

• Vector search finds passages that mean something similar, even with different words.
• Keyword search finds exact terms, which matters for names, codes and product identifiers.
• Hybrid search combines both, and is often the best default.

Most RAG failures are retrieval failures. If the right passage is never found, the model has nothing to work from.

What usually goes wrong

• Chunks are the wrong size. Too small loses context, and too large dilutes the match.
• The data is stale or messy. Old versions and drafts get retrieved next to the current policy.
• The right passage is not found, because the question and the document use different wording.
• The model ignores the passage or blends it with its own guesses.
• Nothing is measured, so problems go unnoticed until a customer finds one.

How to measure a RAG system

Build a small evaluation set: real questions with the passage that should answer each. Then measure two things separately:

1. Retrieval quality: was the right passage in the results?
2. Answer quality: given the passages, was the answer correct, relevant and grounded in them?

The AI-103 guide names evaluating "fabrications, relevance, quality, and safety" and monitoring "data ingestion quality, search index health, and relevance performance". Separating the two measurements tells you which half to fix.

Security and safety

Retrieved text is untrusted input. A document in your index could contain instructions aimed at the model, which is an indirect form of prompt injection. The OWASP Top 10 for LLM Applications (2025) lists Prompt Injection, Vector and Embedding Weaknesses, and Misinformation among its risks, all of which touch RAG. Practical steps include controlling who can add documents, filtering what is retrieved by the user's permissions, and never letting retrieved text trigger actions without checks. See AI guardrails and safety.

RAG and agents

RAG is often one tool inside a larger system. An agent might decide when to search, which index to search and whether the results are good enough. Understanding RAG first makes agents much easier.

Where RAG appears in certifications

RAG is named directly in the AI-103 outline, and the AWS AIF-C01 exam covers the surrounding decision of whether to prompt, retrieve or fine-tune. The full route is in the certification roadmap.

Learn it by building one

The best way to understand RAG is to build a tiny one over documents you know, then ask questions you know the answers to and see where it fails. That is the approach behind learning by doing. Code Nexus has hands-on episodes on grounding an assistant that invents answers, where you find out why it went wrong and fix it. Create a free account to start with the fundamentals.