Introduction

A language model learns from text — billions of sentences compressed into billions of numbers called weights. Ask it a question and it answers by pattern-matching against everything it absorbed during training. That sounds powerful, and often it is. But it has a silent flaw: the weights are frozen.

Once training ends, the model's knowledge is fixed. Ask about a news story from last week and it has nothing to say. Ask about your company's internal documents and it cannot possibly know them. The model can only confabulate — invent plausible-sounding text with no real source to back it up.

Retrieval-Augmented Generation (RAG) was introduced in 2020 by Lewis et al. (Facebook AI Research) to fix exactly this. Instead of generating from memory alone, the model first retrieves relevant passages from an external corpus, then generates an answer that is grounded in those passages. The weights provide language understanding; the corpus provides up-to-date, verifiable facts.

The idea is elegant because it separates two very different problems: what to say (the retrieved documents) from how to say it (the language model). Update the corpus and the answers update — no retraining required.

Try It

Below is a tiny corpus of four short documents. Pick a question, then click Retrieve & Answer. The demo scores every document against your query (using term-overlap, a simplified BM25), highlights the top match, and composes a grounded answer from it.

<!-- RAG demo: two-stage retrieve-then-generate on a tiny 4-document corpus -->
<div class="rag-app">
  <div class="corpus-section">
    <div class="section-label">Corpus</div>
    <div id="doc-list" class="doc-list"></div>
  </div>
  <div class="query-section">
    <div class="section-label">Query</div>
    <div class="query-row">
      <select id="query-select"></select>
      <button id="run-btn" type="button">Retrieve & Answer</button>
    </div>
  </div>
  <div class="result-section" id="result-section" style="display:none">
    <div class="section-label">Retrieval + Generation</div>
    <div class="scores-row" id="scores-row"></div>
    <div class="answer-box" id="answer-box"></div>
  </div>
</div>

/* Styles for the RAG demo: corpus cards, query row, score chips, answer box */
* { box-sizing: border-box; }
body { font-family: system-ui, sans-serif; color: #222; margin: 0; font-size: 14px; }
.rag-app { display: flex; flex-direction: column; gap: .9rem; padding: .2rem; }
.section-label { font-size: .72rem; font-weight: 700; text-transform: uppercase;
  letter-spacing: .06em; color: #5a7088; margin-bottom: .35rem; }
/* Document cards — highlighted green when top match */
.doc-list { display: grid; grid-template-columns: repeat(2, 1fr); gap: .5rem; }
.doc-card { background: #e8eef3; border: 1.5px solid #cdd9e3; border-radius: 8px;
  padding: .5rem .65rem; font-size: .82rem; line-height: 1.45; cursor: default;
  transition: border-color .15s, background .15s; }
.doc-card.top-match { background: #d4edda; border-color: #28a745; }
.doc-card .doc-id { font-weight: 700; font-size: .7rem; color: #5a7088; margin-bottom: .18rem; }
/* Query selector + run button */
.query-row { display: flex; gap: .5rem; flex-wrap: wrap; }
select { flex: 1; min-width: 0; padding: .4rem .6rem; border: 1.5px solid #cdd9e3;
  border-radius: 8px; font: 14px system-ui, sans-serif; background: #fff; color: #222; }
button { padding: .4rem .9rem; background: #1d3557; color: #fff; border: none;
  border-radius: 8px; font: 600 14px system-ui, sans-serif; cursor: pointer; white-space: nowrap; }
button:hover { background: #162a43; }
/* BM25 score chips per document */
.scores-row { display: flex; gap: .4rem; flex-wrap: wrap; margin-bottom: .4rem; }
.score-chip { font-size: .75rem; padding: .18rem .5rem; border-radius: 999px;
  background: #e8eef3; border: 1px solid #cdd9e3; }
.score-chip.best { background: #28a745; color: #fff; border-color: #28a745; }
/* Grounded answer panel */
.answer-box { background: #fff8e1; border: 1.5px solid #f0b429; border-radius: 8px;
  padding: .55rem .75rem; font-size: .9rem; line-height: 1.5; }
.answer-box .ans-label { font-size: .7rem; font-weight: 700; text-transform: uppercase;
  letter-spacing: .06em; color: #b08000; margin-bottom: .25rem; }

const fmt = (s, v) => s.replace(/%(\w+)%/g, (_, k) => String(v[k] ?? ''));
// RAG (Retrieval-Augmented Generation) demo — simplified BM25 retrieval
// Step 1: score every document against the query (retrieval)
// Step 2: compose a grounded answer from the top-scoring document (generation)

const CORPUS = [
  { id: "D1", text: "Transformers use self-attention to weigh every token against every other token, enabling long-range dependencies in text." },
  { id: "D2", text: "RAG grounds language model outputs in retrieved documents, reducing hallucination and keeping knowledge up to date without retraining." },
  { id: "D3", text: "PageRank assigns importance to web pages by counting how many high-quality pages link to them, giving a recursive authority score." },
  { id: "D4", text: "BM25 is a bag-of-words ranking function that scores documents by term frequency, adjusted for document length and a saturation parameter." }
];

const QUERIES = [
  "How does RAG reduce hallucination?",
  "What is BM25 and how does it rank documents?",
  "How do transformers handle long-range dependencies?",
  "How does PageRank measure page importance?"
];

// Tokenize: lowercase, strip punctuation, split on whitespace
function tokenize(str) {
  return str.toLowerCase().replace(/[^a-z0-9áéíóúüñ\s]/g, '').split(/\s+/).filter(Boolean);
}

// Simplified BM25 term-frequency scoring (no IDF — corpus too small)
function bm25Score(queryTokens, docTokens) {
  const k1 = 1.5, b = 0.75;
  const avgLen = 18; // approximate average document length in tokens
  const tf = {};
  docTokens.forEach(t => { tf[t] = (tf[t] || 0) + 1; });
  let score = 0;
  queryTokens.forEach(qt => {
    const f = tf[qt] || 0;
    if (f > 0) {
      score += (f * (k1 + 1)) / (f + k1 * (1 - b + b * docTokens.length / avgLen));
    }
  });
  return score;
}

// Compose a grounded answer quoting the source document
function groundedAnswer(query, doc) {
  return fmt('Based on document "%id%": %text%', { id: doc.id, text: doc.text });
}

function renderCorpus(highlightId) {
  const list = document.getElementById('doc-list');
  list.innerHTML = '';
  CORPUS.forEach(doc => {
    const card = document.createElement('div');
    card.className = 'doc-card' + (doc.id === highlightId ? ' top-match' : '');
    card.innerHTML = '<div class="doc-id">' + doc.id + '</div>' + doc.text;
    list.appendChild(card);
  });
}

function populateQueries() {
  const sel = document.getElementById('query-select');
  sel.innerHTML = '';
  QUERIES.forEach(q => {
    const opt = document.createElement('option');
    opt.value = q;
    opt.textContent = q;
    sel.appendChild(opt);
  });
}

function run() {
  const query = document.getElementById('query-select').value;
  const qTokens = tokenize(query);

  // Score all documents and sort descending
  const scored = CORPUS.map(doc => ({
    doc,
    score: bm25Score(qTokens, tokenize(doc.text))
  })).sort((a, b) => b.score - a.score);

  const best = scored[0];

  // Show score chips for every document
  const scoresRow = document.getElementById('scores-row');
  scoresRow.innerHTML = scored.map(s =>
    '<span class="score-chip' + (s.doc.id === best.doc.id ? ' best' : '') + '">' +
    s.doc.id + ': ' + s.score.toFixed(2) + '</span>'
  ).join('');

  renderCorpus(best.doc.id);

  // Show grounded answer from the best document
  const answerBox = document.getElementById('answer-box');
  answerBox.innerHTML = fmt('<div class="ans-label">Grounded answer</div>%groundedAnswer%', { groundedAnswer: groundedAnswer(query, best.doc) });

  document.getElementById('result-section').style.display = 'block';
}

document.getElementById('run-btn').addEventListener('click', run);
populateQueries();
renderCorpus(null);
Result
Console

Notice the two-stage structure: retrieval is fast keyword scoring across all documents; generation assembles a sentence from the winning passage. Change the query or edit a document and the answer changes immediately — no model retraining, just a new search.

The Real Complexity

RAG sounds simple — search, then generate — but each stage hides real difficulty:

  • Indexing: raw documents must be split into chunks small enough for the retriever to score meaningfully but large enough to hold useful context. Too small and chunks lose context; too large and the signal drowns in noise.
  • Retrieval quality: classic keyword search (BM25) is fast but misses synonyms and paraphrases. Dense retrieval — encoding chunks and queries into high-dimensional vectors, then finding nearest neighbors — captures semantics but requires a learned embedding model and a fast approximate nearest-neighbor index (e.g., HNSW or FAISS).
  • Faithfulness: the generator must stay close to the retrieved text. A powerful language model can still "hallucinate" details not in the passage. Measuring and enforcing faithfulness is an active research problem.
  • Latency: retrieval adds a round-trip — indexing the corpus offline, then at query time scoring millions of vectors in milliseconds. Production systems invest heavily in approximate indexing and caching.

Compared to a transformer that must compress all knowledge into its weights at training time, RAG shifts the hard problem from memorization to retrieval quality. It is an engineering trade-off, not a free lunch.

Where It Matters

RAG is the architecture of choice wherever accuracy and traceability matter more than creative fluency:

  • Enterprise Q&A: companies index internal wikis, Slack archives, and policy documents so employees can ask questions and receive answers with citations — not hallucinated guesses.
  • Legal and medical research: retrieval over case-law or clinical guidelines ensures that every claim links to a real source that can be audited.
  • Customer support: a RAG system over product documentation answers support tickets far more accurately than a purely generative chatbot that might invent features.
  • Scientific literature: researchers use RAG to query thousands of papers and get synthesized, cited summaries — the same pattern behind tools like Consensus and Perplexity.
  • Long-tail knowledge: niche topics underrepresented in training data are served by a targeted corpus, not by hoping the model happened to learn them.

RAG also pairs naturally with transformer architectures: the retriever and generator are often both transformers, fine-tuned jointly or kept separate depending on how dynamic the corpus needs to be. The PageRank idea of ranking by authority echoes in how RAG systems weight retrieved passages.

Conclusion

Language models are extraordinary pattern-matchers, but they cannot remember what they never saw and cannot update what they memorized years ago. Retrieval-Augmented Generation solves this by separating the roles cleanly: the corpus holds the facts, the model holds the language.

The result is an AI assistant that can cite its sources, update its knowledge by swapping a corpus, and fail gracefully when no document supports the answer — rather than confidently making something up.

That combination of power and traceability is why RAG has become the default architecture for any system where being wrong costs something: medical advice, legal research, financial analysis, or simply a support bot that must not invent features. The hard part was never generating fluent text — it was having something true to say.

Share this article

Pick a channel — or use your device's native share sheet.

Comments

Loading comments...

https://www.kipuhub.com/en/article/retrieval-augmented-generation/Content licensed under CC BY-NC 4.0.