← All artifacts · All patterns · Retrieval & RAG

Query Rewriting

aka Multi-Query Retrieval, Query Expansion, Query Reformulation, RAG-Fusion (query side)

category: retrieval · status: mature

Use an LLM to generate several alternative formulations of the user's query, retrieve documents for each, and rank-fuse the results so recall does not depend on one phrasing. A documentation assistant gets the query 'why is my deploy slow?' A single embedding lookup retrieves a handful of chunks about deploy speed in general. The rewriter produces variants — 'deployment latency causes', 'slow CI pipeline diagnosis', 'docker image build time bottlenecks', 'kubernetes rollout delay reasons' — and runs retrieval for each in parallel. The fused result set now contains chunks about image-build caching and rollout strategy that the original query embedding sat too far from. Reciprocal Rank Fusion across the five rankings surfaces the chunks that appear in multiple lists.

2. Problem

A team runs retrieval over a corpus where the user's natural phrasing is only one of many ways to express the same information need. The corpus chunks may use different vocabulary, abbreviations, or framing for the same concept, and an embedding-based lookup against a single query vector lands in only one neighbourhood of the embedding space. Users themselves under-specify, ask compound questions, or use idioms the corpus does not echo.

A single query embedding samples only one point in the semantic space and retrieves only the chunks closest to that point. Relevant chunks expressed in different vocabulary, at a different specificity level, or framed as a different sub-question are missed entirely, and no downstream reranker can rescue a chunk that was never retrieved. The user's first phrasing is a noisy estimator of intent, and recall is bottlenecked by how well that one phrasing aligns with how the answer chunks were written.

Competing forces:

• More query variants improve recall but multiply retrieval cost linearly.
• Variants generated by the LLM may drift off-topic and inject noise into the result set.
• Fusion strategy (union, RRF, weighted) decides whether rare-but-relevant chunks survive deduplication.
• Latency budget bounds how many parallel retrievals the system can afford per request.

3. When to Use

• Users under-specify or phrase queries in vocabulary that does not echo the corpus.
• Queries are compound and naturally decompose into sub-questions answered by different chunks.
• Single-query retrieval shows recall@k below the level a reranker could rescue.
• Latency budget can absorb one LLM call plus N parallel retrievals.

4. When Not to Use

• Queries are short and exact (codes, identifiers, exact-match lookups) where variants would only add noise.
• Sub-100ms p95 retrieval is required and the extra LLM call blows the budget.
• Corpus is small enough that a single embedding lookup already saturates recall.

5. Architecture Diagram

flowchart TD
  Q[Original query] --> R[LLM rewriter]
  R --> V1[Variant 1]
  R --> V2[Variant 2]
  R --> V3[Variant N]
  Q --> V0[Original kept as variant 0]
  V0 --> Ret[Retriever]
  V1 --> Ret
  V2 --> Ret
  V3 --> Ret
  Ret --> F[Rank fusion / RRF]
  F --> Out[Top-N to generator or reranker]

Query Rewriting expands the user query into N variants, retrieves for each in parallel, and rank-fuses the results.

6. Components

• Query rewriter — LLM call that emits N variants of the user query in a structured format
• Parallel retriever — runs the underlying retriever (dense, sparse, or hybrid) once per variant
• Rank fusion stage — Reciprocal Rank Fusion or deduplicated union merges the N ranked lists into one
• Top-N selector — emits the fused top-N to the generator or to a downstream reranker

7. Tools

• LLM rewriter — small fast model is sufficient (e.g. Haiku-class); the rewriter does not need the strongest model
• Retriever — any existing retriever (vector index, BM25, or hybrid) reused unchanged per variant
• Fusion library — RRF implementation or vector-store SDK that exposes hybrid/multi-query helpers

8. Guardrails

Therefore: have the LLM rewrite the query into several variants, retrieve for each in parallel, and fuse the rankings so the answer set reflects the intent rather than the surface phrasing.

9. Failure Modes

• Drifting variant — one rewrite shifts the topic, its results enter the fused list, and distractors push relevant chunks down.
• Lost original — the original query is not kept as a variant, and the system does worse than plain retrieval on queries that were already well phrased.
• Fusion drops the rare hit — a relevant chunk found by only one variant ranks too low after fusion and is cut from the top-N.
• Latency multiplier — the rewrite call plus N parallel retrievals push a simple lookup past the request's time budget.
• Retrieval cost and latency multiply by the number of variants.
• LLM-generated variants can drift off-topic and inject distractors into the result set.
• Fusion tuning (RRF constant, weight, union policy) is empirical and corpus-specific.
• An extra LLM call sits on the request path before any retrieval can start.

10. Evaluation Metrics

• Recall@k against single-query baseline — quantifies the lift from variant expansion
• Per-variant contribution — fraction of fused top-N chunks unique to each variant, to detect dead variants
• Distractor rate — fraction of retrieved chunks that the generator does not cite, to detect off-topic drift
• p95 retrieval latency — overhead of the rewriter LLM call plus N parallel retrievers
• Cost per query — extra LLM call plus N retrievals against the single-query baseline

11. Code Examples

function multi_query_retrieve(query, n=4, k=10):
    variants = llm.generate_variants(query, n=n)         # paraphrase / decompose / step-back
    variants = [query] + variants                        # keep the original

    rankings = []
    for v in variants:
        rankings.append(retriever.search(v, k=k))        # parallel

    # Reciprocal Rank Fusion across variant rankings
    C = 60
    scores = {}
    for ranking in rankings:
        for rank, doc in enumerate(ranking):
            scores[doc.id] = scores.get(doc.id, 0) + 1 / (C + rank)

    fused = sort_desc(scores.items(), by=value)
    return [docs[doc_id] for doc_id, _ in fused[:k]]

import logging
from langchain.chat_models import init_chat_model
from langchain_classic.retrievers.multi_query import MultiQueryRetriever

llm = init_chat_model("gpt-4o-mini")
base_retriever = vector_store.as_retriever(search_kwargs={"k": 5})

retriever = MultiQueryRetriever.from_llm(
    retriever=base_retriever,
    llm=llm,
)

# Surface the generated variants in logs.
logging.basicConfig()
logging.getLogger("langchain.retrievers.multi_query").setLevel(logging.INFO)

docs = retriever.invoke("What is task decomposition for LLM agents?")
for d in docs:
    print(d.page_content[:200])

from llama_index.core import VectorStoreIndex
from llama_index.core.indices.query.query_transform import HyDEQueryTransform
from llama_index.core.query_engine import TransformQueryEngine
from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-4o-mini")
index = VectorStoreIndex.from_documents(documents)
base_query_engine = index.as_query_engine(llm=llm, similarity_top_k=5)

# HyDE-style query transformation: the LLM rewrites the short query
# into an answer-shaped paragraph that drives the retriever.
hyde = HyDEQueryTransform(llm=llm, include_original=True)
query_engine = TransformQueryEngine(
    query_engine=base_query_engine,
    query_transform=hyde,
)

response = query_engine.query("What is task decomposition?")
print(response)

function multi_query_retrieve(query, n=4, k=10):
    variants = llm.generate_variants(query, n=n)   # paraphrase / decompose
    variants = [query] + variants                  # keep the original

    rankings = parallel_map(
        lambda v: retriever.search(v, k=k),
        variants,
    )

    # Reciprocal Rank Fusion across variant rankings (rank-based, scale-free).
    C = 60
    scores = {}
    for ranking in rankings:
        for rank, doc in enumerate(ranking):
            scores[doc.id] = scores.get(doc.id, 0) + 1 / (C + rank)

    fused    = sort_desc(scores.items(), by=value)
    top      = [docs[doc_id] for doc_id, _ in fused[:k]]
    reranked = reranker.rerank(query, top)
    return reranked

function answer(query):
    docs = multi_query_retrieve(query, n=4, k=10)
    return llm.generate(prompt=query, context=docs[:5])

# Why this works: the user's first phrasing is a noisy estimator of
# intent; retrieving against several formulations samples more of the
# semantic space, and rank-based fusion keeps the BM25/dense/variant
# scales from contaminating one another.
# Related: see `modular-rag`, `hierarchical-retrieval`.