← All artifacts · All patterns · Memory

Vector Memory

aka Embedding-Indexed Memory, Vector Store Memory

category: memory · status: mature

Store memories as embeddings in a vector index and retrieve the most semantically similar items at query time. A long-running personal agent's append-only thought log grows past a million entries; finding relevant past becomes hopeless and dumping it all into context is impossible. The team embeds each memory item, indexes it in a vector store, and at query time retrieves top-k semantically similar items (plus optional recency boost). Now 'what did I decide about latency three months ago' returns the actual right entries rather than the most recent or none, and prompt size stays bounded as memory grows.

2. Problem

A long-running agent accumulates facts and observations over time, and on each step it needs to find the small subset of past items that is relevant to the current situation. Relevance is best judged by semantic similarity rather than by exact term match or chronological recency: 'find the past notes whose meaning is close to what is happening now'.

An append-only log of everything the agent has seen grows unboundedly and quickly becomes too large to search by linear scan. Without a semantic retrieval layer, the agent has no way to find the relevant past, because keyword search misses paraphrase and chronological recency misses older but topically relevant items. The team needs a memory store that supports similarity queries against an embedding of the current context, so that the agent can pull back exactly the items it should be thinking about now.

Competing forces:

• Embedding choice constrains retrieval quality.
• Index updates have non-trivial latency.
• Forgetting is achieved by deletion or decay; both have failure modes.

3. When to Use

• Long-running agents accumulate facts whose relevance is best judged by similarity.
• Append-only logs would otherwise grow unboundedly without retrieval.
• An embedding model and vector index can be deployed and maintained.

4. When Not to Use

• Memory is small and a typed key-value store would serve better.
• Recency or exact-match retrieval matters more than semantic similarity.
• Vector index maintenance cost outweighs the retrieval benefit.

5. Architecture Diagram

flowchart TD
  Mem[New memory item] --> Emb[Embed]
  Emb --> Idx[(Vector index)]
  Q[Query / current state] --> QEmb[Embed]
  QEmb --> Top[Retrieve top-k similar]
  Idx --> Top
  Top --> Decay[Apply decay / salience weighting]
  Decay --> Ctx[Prepend to context]

6. Components

• Embedding model — produces a vector for each memory item at write time and for the query at read time
• Vector index — persistent ANN store holding (id, vector, payload) tuples for similarity search
• Writer — embeds new memory items and upserts them into the index
• Retriever — embeds the query or current state, fetches top-k nearest neighbours, returns payloads
• Decay/salience weighter — adjusts neighbour scores with recency boost or salience to break ties on raw similarity

7. Tools

• Vector database — Pinecone, Weaviate, Chroma, FAISS, or pgvector for the index itself
• Embedding model — OpenAI text-embedding-3, Cohere embed-v3, or a local bge model
• Optional reranker — cross-encoder pass over the top-k to lift retrieval quality before context injection

8. Guardrails

Therefore: embed every memory item and retrieve the top-k most similar at query time, so that recall is driven by semantic match instead of exact keys or scrollback position.

9. Failure Modes

• Temporal blind spot — a question like what happened yesterday returns similar items from any date, not the recent ones.
• Embedding drift — the embedding model changes, and old vectors no longer match new queries well.
• Near-miss retrieval — the top-k items share wording with the query but not its meaning, and the agent reasons from the wrong memories.
• Decay mistuned — aging is too strong and older relevant items never surface, or too weak and stale items crowd out new ones.
• Misses purely temporal queries ('what did I do yesterday?').
• Embedding drift on schema changes.

10. Evaluation Metrics

• Recall@k of golden memories — fraction of labelled relevant items that appear in the top-k for matching queries
• Retrieval latency p95 — wall-clock for embed+query+rerank on a typical store size
• Embedding-drift incidents — count of breakages caused by changing the embedding model without reindexing
• Temporal-query miss rate — fraction of 'what did I do yesterday' style questions where similarity alone fails
• Storage and index cost per million items — the operational floor under which the pattern stops paying off

11. Code Examples

def remember(text, meta):
    vec = embed(text)
    index.add(id=uuid(), vector=vec, payload={"text": text,
                                              "ts": now(), **meta})

def recall(query, k=5, decay_half_life_days=30):
    q = embed(query)
    hits = index.search(q, k=k * 3)             # over-fetch for reranking
    for h in hits:
        age_days = (now() - h.payload["ts"]) / 86400
        # Older items decay; salience boost from caller-provided meta.
        h.score *= 0.5 ** (age_days / decay_half_life_days)
        h.score *= h.payload.get("salience", 1.0)
    hits.sort(key=lambda h: h.score, reverse=True)
    return [h.payload["text"] for h in hits[:k]]

from mem0 import Memory

m = Memory()  # default: local vector store + LLM-driven distillation

# Each .add embeds the distilled fact and writes it into Alice's vector
# namespace. The raw turn is summarised first, then indexed.
m.add(
    [
        {"role": "user",      "content": "I'm allergic to peanuts."},
        {"role": "assistant", "content": "Noted."},
    ],
    user_id="alice",
)
m.add(
    [
        {"role": "user",      "content": "My favourite cuisine is Thai."},
        {"role": "assistant", "content": "Got it."},
    ],
    user_id="alice",
)

# Semantic recall — the query embedding finds the allergy note even
# though the user phrased it differently.
results = m.search("any food restrictions?", user_id="alice", limit=3)
for hit in results.get("results", []):
    print(hit["score"], hit["memory"])

import os
from letta_client import Letta

client = Letta(api_key=os.getenv("LETTA_API_KEY"))

agent = client.agents.create(
    model="openai/gpt-4o-mini",
    memory_blocks=[
        {"label": "persona", "value": "travel planner", "limit": 1000},
    ],
)

# Write: each passage is embedded with the agent's configured embedding
# model and stored in the archival vector index.
client.agents.passages.insert(
    agent_id=agent.id,
    content="Alice is allergic to peanuts.",
    tags=["profile", "diet"],
)
client.agents.passages.insert(
    agent_id=agent.id,
    content="Alice's favourite cuisine is Thai.",
    tags=["profile", "diet"],
)

# Read: vector search over the archival index, optionally filtered by tag.
hits = client.agents.passages.search(
    agent_id=agent.id,
    query="food restrictions",
    tags=["diet"],
)
for p in hits:
    print(p.content)