Retrieval-Augmented Generation

A language model learns from text — billions of sentences compressed into billions of numbers called weights. Ask it a question and it answers by pattern-matching against everything it absorbed during training. That sounds powerful, and often it is. But it has a silent flaw: the weights are frozen.

Once training ends, the model's knowledge is fixed. Ask about a news story from last week and it has nothing to say. Ask about your company's internal documents and it cannot possibly know them. The model can only confabulate — invent plausible-sounding text with no real source to back it up.

Retrieval-Augmented Generation (RAG) was introduced in 2020 by Lewis et al. (Facebook AI Research) to fix exactly this. Instead of generating from memory alone, the model first retrieves relevant passages from an external corpus, then generates an answer that is grounded in those passages. The weights provide language understanding; the corpus provides up-to-date, verifiable facts.

The idea is elegant because it separates two very different problems: what to say (the retrieved documents) from how to say it (the language model). Update the corpus and the answers update — no retraining required.

Below is a tiny corpus of four short documents. Pick a question, then click Retrieve & Answer. The demo scores every document against your query (using term-overlap, a simplified BM25), highlights the top match, and composes a grounded answer from it.

<!-- {{c_html_comment}} -->
<div class="rag-app">
  <div class="corpus-section">
    <div class="section-label">{{label_corpus}}</div>
    <div id="doc-list" class="doc-list"></div>
  </div>
  <div class="query-section">
    <div class="section-label">{{label_query}}</div>
    <div class="query-row">
      <select id="query-select"></select>
      <button id="run-btn" type="button">{{btn_run}}</button>
    </div>
  </div>
  <div class="result-section" id="result-section" style="display:none">
    <div class="section-label">{{label_result}}</div>
    <div class="scores-row" id="scores-row"></div>
    <div class="answer-box" id="answer-box"></div>
  </div>
</div>

/* {{c_css_comment}} */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; font-size: 14px; }
.rag-app { display: flex; flex-direction: column; gap: .9rem; padding: .2rem; }
.section-label { font-size: .72rem; font-weight: 700; text-transform: uppercase;
  letter-spacing: .06em; color: #5a7088; margin-bottom: .35rem; }
/* {{c_doc_cards}} */
.doc-list { display: grid; grid-template-columns: repeat(2, 1fr); gap: .5rem; }
.doc-card { background: #e8eef3; border: 1.5px solid #cdd9e3; border-radius: 8px;
  padding: .5rem .65rem; font-size: .82rem; line-height: 1.45; cursor: default;
  transition: border-color .15s, background .15s; }
.doc-card.top-match { background: #d4edda; border-color: #28a745; }
.doc-card .doc-id { font-weight: 700; font-size: .7rem; color: #5a7088; margin-bottom: .18rem; }
/* {{c_query_row}} */
.query-row { display: flex; gap: .5rem; flex-wrap: wrap; }
select { flex: 1; min-width: 0; padding: .4rem .6rem; border: 1.5px solid #cdd9e3;
  border-radius: 8px; font: 14px system-ui, sans-serif; background: #fff; color: #222; }
button { padding: .4rem .9rem; background: #1d3557; color: #fff; border: none;
  border-radius: 8px; font: 600 14px system-ui, sans-serif; cursor: pointer; white-space: nowrap; }
button:hover { background: #162a43; }
/* {{c_scores_row}} */
.scores-row { display: flex; gap: .4rem; flex-wrap: wrap; margin-bottom: .4rem; }
.score-chip { font-size: .75rem; padding: .18rem .5rem; border-radius: 999px;
  background: #e8eef3; border: 1px solid #cdd9e3; }
.score-chip.best { background: #28a745; color: #fff; border-color: #28a745; }
/* {{c_answer_box}} */
.answer-box { background: #fff8e1; border: 1.5px solid #f0b429; border-radius: 8px;
  padding: .55rem .75rem; font-size: .9rem; line-height: 1.5; }
.answer-box .ans-label { font-size: .7rem; font-weight: 700; text-transform: uppercase;
  letter-spacing: .06em; color: #b08000; margin-bottom: .25rem; }

// Code not found

Notice the two-stage structure: retrieval is fast keyword scoring across all documents; generation assembles a sentence from the winning passage. Change the query or edit a document and the answer changes immediately — no model retraining, just a new search.

RAG sounds simple — search, then generate — but each stage hides real difficulty:

• Indexing: raw documents must be split into chunks small enough for the retriever to score meaningfully but large enough to hold useful context. Too small and chunks lose context; too large and the signal drowns in noise.
• Retrieval quality: classic keyword search (BM25) is fast but misses synonyms and paraphrases. Dense retrieval — encoding chunks and queries into high-dimensional vectors, then finding nearest neighbors — captures semantics but requires a learned embedding model and a fast approximate nearest-neighbor index (e.g., HNSW or FAISS).
• Faithfulness: the generator must stay close to the retrieved text. A powerful language model can still "hallucinate" details not in the passage. Measuring and enforcing faithfulness is an active research problem.
• Latency: retrieval adds a round-trip — indexing the corpus offline, then at query time scoring millions of vectors in milliseconds. Production systems invest heavily in approximate indexing and caching.

Compared to a transformer that must compress all knowledge into its weights at training time, RAG shifts the hard problem from memorization to retrieval quality. It is an engineering trade-off, not a free lunch.

RAG is the architecture of choice wherever accuracy and traceability matter more than creative fluency:

• Enterprise Q&A: companies index internal wikis, Slack archives, and policy documents so employees can ask questions and receive answers with citations — not hallucinated guesses.
• Legal and medical research: retrieval over case-law or clinical guidelines ensures that every claim links to a real source that can be audited.
• Customer support: a RAG system over product documentation answers support tickets far more accurately than a purely generative chatbot that might invent features.
• Scientific literature: researchers use RAG to query thousands of papers and get synthesized, cited summaries — the same pattern behind tools like Consensus and Perplexity.
• Long-tail knowledge: niche topics underrepresented in training data are served by a targeted corpus, not by hoping the model happened to learn them.

RAG also pairs naturally with transformer architectures: the retriever and generator are often both transformers, fine-tuned jointly or kept separate depending on how dynamic the corpus needs to be. The PageRank idea of ranking by authority echoes in how RAG systems weight retrieved passages.

Language models are extraordinary pattern-matchers, but they cannot remember what they never saw and cannot update what they memorized years ago. Retrieval-Augmented Generation solves this by separating the roles cleanly: the corpus holds the facts, the model holds the language.

The result is an AI assistant that can cite its sources, update its knowledge by swapping a corpus, and fail gracefully when no document supports the answer — rather than confidently making something up.

That combination of power and traceability is why RAG has become the default architecture for any system where being wrong costs something: medical advice, legal research, financial analysis, or simply a support bot that must not invent features. The hard part was never generating fluent text — it was having something true to say.