← All artifacts · All patterns · Retrieval & RAG

Naive RAG

aka Retrieval-Augmented Generation, Top-K Retrieve-and-Stuff

category: retrieval · status: mature

Condition the generator on top-k chunks retrieved from an external dense index so knowledge lives outside parameters. A startup ships a support assistant whose knowledge changes weekly — release notes, pricing, integration guides. Bake-it-into-the-prompt does not scale and fine-tuning on every release is impractical. They adopt naive-rag: chunk the docs, embed with a dense encoder, index, and at query time retrieve top-k and prepend to the prompt. The pipeline is the simplest possible and ships in a week. Knowledge updates now flow by re-indexing the docs, not by retraining or redeploying the model.

2. Problem

A team needs a model to answer questions whose answers depend on information that lives in a corpus too large to fit into the prompt — internal documentation, a knowledge base, a product catalogue, recent news, a body of research papers. The corpus also changes regularly, faster than retraining the base model would allow, so any answers based on the model's training data alone will go stale or be missing entirely.

A bare language model has no access to information beyond what is baked into its weights, and any attempt to answer from parametric memory alone tends to hallucinate plausible-sounding answers, cannot cite a source, and cannot be updated without retraining. The team needs the model to pull relevant external knowledge in at query time, but doing so requires deciding how to chunk the corpus, how to index it, what to retrieve per query, and how to feed it into the prompt. Without that retrieval machinery, the model is stuck with what it already knew at training time.

Competing forces:

• Chunk size trades context loss for retrieval recall.
• Embedding choice constrains retrieval quality.
• Single-shot retrieval misses multi-hop questions.

3. When to Use

• Knowledge lives outside the model and must be conditioned on at query time.
• Citations must be tied to retrieved sources, not invented from parameters.
• A simple chunk-and-embed pipeline meets the recall and quality bar.

4. When Not to Use

• The needed knowledge is already in a tool, database, or scoped system prompt (see naive-rag-first).
• Global, corpus-wide questions need GraphRAG or hierarchical retrieval instead.
• Chunk-level retrieval is the wrong shape for the queries you actually serve.

5. Architecture Diagram

flowchart TD
  C[Corpus] -->|chunk + embed| I[(Vector index)]
  Q[Query] -->|embed| E[Query vector]
  E -->|top-k| I
  I -->|chunks| P[Prompt]
  Q --> P
  P --> G[Generator LLM]
  G --> A[Answer]

6. Components

• Chunker — splits the corpus into retrieval-sized passages at index time
• Embedding encoder — dense model that vectorises both chunks and queries
• Vector index — stores chunk vectors and serves top-k nearest-neighbour lookups
• Prompt assembler — prepends retrieved chunks to the user query before generation
• Generator LLM — answers conditioned on the retrieved chunks plus its parametric memory

7. Tools

• Embedding model — dense encoder for chunks and queries
• Vector index — approximate nearest-neighbour store such as FAISS, Pinecone, or pgvector
• LLM API — generator that consumes the chunk-augmented prompt

8. Guardrails

Therefore: chunk-embed the corpus, retrieve top-k by similarity to the query embedding, and condition generation on the retrieved chunks, so that knowledge lives outside the model parameters.

9. Failure Modes

• Severed context — the chunk boundary cuts a fact away from the sentence that names its subject, and the chunk that holds the answer never matches the query.
• Distractor in the top-k — a chunk that shares the query's words but not its answer ranks high, and the generator answers from it.
• Multi-hop miss — the answer needs two chunks linked by an entity the query never mentions, and one retrieval pass returns only the first.
• Parametric override — retrieval returns nothing useful, and the generator answers from memory with no sign that the retrieved set was empty or weak.
• Chunk boundaries destroy context.
• Top-k retrieval is recall-oriented; precision suffers without reranking.
• No iterative retrieval; multi-hop fails.

10. Evaluation Metrics

• Answer faithfulness against retrieved chunks — share of claims supported by the prepended evidence
• Recall@k of the gold chunk — does the retriever surface the document that contains the answer
• Hallucination rate when retrieval misses — fallback-to-parametric failure mode this pattern is supposed to reduce
• p95 retrieval latency — single-shot retrieval budget that downstream patterns build on
• Cost per answered query — combined embedding, index, and generator spend

11. Code Examples

# Index time (run once)
function build_index(docs):
    chunks = [c for d in docs for c in split(d)]
    for c in chunks:
        c.embedding = embed(c.text)
    return vector_index.add(chunks)

# Query time
function answer(question, k=4):
    query_vec = embed(question)
    chunks    = vector_index.search(query_vec, k=k)
    prompt    = f"Answer using only the context.\n\nContext:\n{join(chunks)}\n\nQuestion: {question}"
    return llm.generate(prompt)

from llama_index.core import SimpleDirectoryReader, VectorStoreIndex

# Load
documents = SimpleDirectoryReader("./data").load_data()

# Index (chunks, embeds, stores)
index = VectorStoreIndex.from_documents(documents)

# Query (retrieves top-k, prepends to prompt, generates)
query_engine = index.as_query_engine()
response = query_engine.query("What is task decomposition?")
print(response)

# Compose naive-rag with chain-of-verification: the first pass is plain
# RAG (retrieve top-k, prepend, generate); a second pass plans verification
# questions over the draft, retrieves separately for each, and revises
# the draft against those grounded answers before delivery.

def rag_with_verification(question, corpus_index):
    # 1. Plain naive-RAG draft.
    q_vec  = embed(question)
    chunks = corpus_index.search(q_vec, k=4)
    draft  = llm.generate(
        prompt=(
            f"Answer using only the context.\n\n"
            f"Context:\n{format_with_ids(chunks)}\n\n"
            f"Question: {question}"
        )
    )

    # 2. Verification plan: factual questions about the draft.
    verify_prompt = (
        "List 3-5 short factual questions whose answers would confirm "
        "or refute claims in this draft. One per line.\n"
        f"DRAFT: {draft}"
    )
    questions = parse_lines(llm.generate(user=verify_prompt))

    # 3. Each verifier gets its OWN fresh retrieval — not the draft's.
    checks = []
    for vq in questions:
        v_docs = corpus_index.search(embed(vq), k=3)
        v_ans = llm.generate(
            system="Answer ONLY from the context. Cite chunk ids as [id].",
            user=f"CONTEXT:\n{format_with_ids(v_docs)}\n\nQ: {vq}",
        )
        checks.append({"q": vq, "a": v_ans,
                        "cites": extract_cites(v_ans)})

    # 4. Revise the draft to be consistent with verified answers.
    revised = llm.generate(
        system="Revise the draft to match the verified answers. Drop "
               "or correct any claim the verification contradicts.",
        user=f"DRAFT: {draft}\nVERIFIED: {format_checks(checks)}",
    )

    return {"answer": revised,
            "sources": collect_cited(chunks, checks),
            "verify":  checks}

# Why this works: bare naive-RAG passes the first retrieval straight into
# the answer with no second look. Composing with CoVe gives each claim
# its own retrieval and forces the draft to match those grounded answers,
# so hallucinations that survived the first pass are caught in the second.
# Related: see `chain-of-verification`, `citation-attribution`, `cross-encoder-reranking`.