aka Personal-Concept Lexicon, Own-Writing Lexicon
Mine a small bounded vocabulary from the agent's own writing and cache it as the conceptual axis for scoring new thoughts, so relevance reflects the agent's actual frame rather than a generic embedding space. An agent has been journalling for three months. Once a week, a mining job aggregates frontmatter tags and high-frequency content tokens across recent thoughts and the long-term insight store, picks the top thirty concepts with weights, and writes them to a small JSON cache. When the agent receives a new thought, the salience scorer combines generic embedding distance to recent context with overlap against the cached vocabulary. A thought that uses three of the top-thirty concepts scores higher than a thought with similar embedding distance but no overlap, because the cached vocabulary says 'this is on the line of inquiry'.
A long-running agent accumulates a corpus of its own output: thought traces, insights, journal entries, notes. Some downstream component wants to score new thoughts for relevance, novelty, or kinship with the agent's existing concerns. The default tool is a generic embedding space, which gives a sensible answer about semantic similarity but tells the agent nothing about its own preoccupations — 'is the agent still pulling at the things it has been pulling at?' is a different question from 'is this semantically close to the previous paragraph?'
Generic embeddings score against the world's distribution of meaning, not the agent's. A new thought that lands inside the agent's persistent web of concerns can come back with the same similarity score as a perfectly off-topic but topically-adjacent one, because the embedding space has no notion of what this particular agent has been writing about for months. The result is a salience signal that is plausible-on-paper and indifferent in practice: the agent cannot tell, from the score alone, whether a thought is on its own line of inquiry or just somewhere in the same neighbourhood.
Competing forces:
flowchart LR Corpus[(Own corpus<br/>thoughts + insights)] Miner[Mining pass<br/>tags + content frequency] Cache[(Top-N vocabulary<br/>cache)] Thought[New thought] Scorer[Salience scorer] Corpus -->|periodic| Miner Miner --> Cache Thought --> Scorer Cache --> Scorer Scorer --> Score[Own-frame score]
Periodic mining derives a self-vocabulary; downstream scoring uses it as an additional axis alongside generic similarity.
Therefore: periodically mine a small top-N concept vocabulary from the agent's own thoughts and insights — using a mix of frontmatter tags and content frequency — cache it to disk, and refresh on a schedule, so scoring new thoughts can use this learned axis alongside generic similarity.
# Periodic miner over the agent's own corpus -> top-N vocabulary cache.
# Downstream scoring blends generic similarity with own-vocab overlap.
import json, math
def mine_vocabulary(corpus_window, top_n=30):
# 1. Collect candidate tokens: frontmatter tags + content tokens.
df_tag = {} # tag frequency across entries
df_token = {} # content-token frequency
N = 0
for entry in corpus_window:
N += 1
for tag in entry.tags:
df_tag[tag] = df_tag.get(tag, 0) + 1
for tok in tokenise(entry.content):
if not is_stopword(tok):
df_token[tok] = df_token.get(tok, 0) + 1
# 2. Blend: tags weighted higher than raw content tokens.
blended = {}
for tag, n in df_tag.items():
blended[tag] = (n / N) * 2.0
for tok, n in df_token.items():
blended[tok] = blended.get(tok, 0) + (n / N)
# 3. Top-N as the cached vocabulary.
top = sorted(blended.items(), key=lambda kv: kv[1], reverse=True)[:top_n]
return {"refreshed_at": now(), "vocab": dict(top)}
def refresh(cache_path, corpus_window):
vocab = mine_vocabulary(corpus_window)
json.dump(vocab, open(cache_path, "w"))
def own_frame_score(thought, vocab):
toks = set(tokenise(thought))
return sum(w for tok, w in vocab["vocab"].items() if tok in toks)
def salience(thought, recent_ctx, vocab, alpha=0.5, beta=0.5):
# Two-axis score: generic similarity + own-frame fit.
return (
alpha * cosine(embed(thought), embed(recent_ctx))
+ beta * own_frame_score(thought, vocab)
)
# Refresh cadence: weekly for stable agents; after each consolidation
# pass for more volatile ones. A reflection step can audit churn vs
# lock-in and adjust the cadence.VOCAB_CACHE = "./cache/self_vocabulary.json"
TOP_N = 200
MINE_WINDOW = "14d" # corpus slice to mine
REFRESH_EVERY = "7d" # cadence; tighter for volatile agents
def mine_self_vocabulary():
corpus = agent_corpus.recent(window=MINE_WINDOW) # thoughts + insights
counts = Counter()
# Two signals: explicit frontmatter tags + content-frequency terms.
for doc in corpus:
for tag in doc.frontmatter.get("tags", []):
counts[tag] += 3 # tags weigh more
for term in extract_content_terms(doc.body):
counts[term] += 1
top = counts.most_common(TOP_N)
vocab = {term: weight for term, weight in top}
memory.write(
key = "self_vocabulary",
value = {"vocab": vocab, "mined_at": now(), "window": MINE_WINDOW},
scope = "agent",
ttl = None, # overwritten on refresh
)
write_disk(VOCAB_CACHE, vocab)
schedule(mine_self_vocabulary, every=REFRESH_EVERY)
# Also: refresh opportunistically after each dream-consolidation cycle.
on_event("dream_consolidation_complete", mine_self_vocabulary)
def score_thought(new_thought, recent_context):
vocab = load_disk(VOCAB_CACHE)
generic = cosine(embed(new_thought), embed(recent_context))
own_axis = overlap_weighted(tokens(new_thought), vocab)
return 0.5 * generic + 0.5 * own_axis # blended salience score
# Why this works: generic embeddings score against the world's distribution
# of meaning, not this agent's. The mined self-vocabulary lets scoring
# distinguish 'on my line of inquiry' from 'merely topically adjacent' —
# a signal embedding-only similarity cannot provide.
# Related: see `vector-memory`, `salience-attention-mechanism`, `dream-consolidation-cycle`.# Compose self-corpus-vocabulary with dream-consolidation-cycle: the
# vocabulary is re-mined at the end of every consolidation cycle, so
# the mined frame and the consolidated insights advance in lockstep.
# Without composition, the vocabulary cache can lag the insight store
# by days or weeks and silently mis-score new thoughts.
VOCAB_KEY = ("self_vocabulary",)
INSIGHTS_KEY = ("insight",)
def dream_consolidation_cycle(agent_id):
# 1. Standard consolidation: mine recurring themes from recent
# thoughts and update the durable insight store.
recent = memory.recall(
query="*", scope=f"agent:{agent_id}",
prefix=("thought",), since=now() - "14d",
)
insights = llm.consolidate(
recent,
instruction="Surface recurring themes, contradictions, and "
"threads that re-appear across distinct sessions.",
)
for ins in insights:
memory.write(
key = INSIGHTS_KEY + (ins.id,),
value = ins,
scope = f"agent:{agent_id}",
ttl = None,
)
# 2. NOW re-mine the vocabulary against the same fresh corpus —
# this is the composition. The vocabulary always reflects the
# state of the insight store, never lags it.
corpus = recent + memory.recall(
query="*", scope=f"agent:{agent_id}", prefix=INSIGHTS_KEY,
)
counts = Counter()
for doc in corpus:
for tag in doc.frontmatter.get("tags", []):
counts[tag] += 3
for term in extract_content_terms(doc.body):
counts[term] += 1
vocab = dict(counts.most_common(200))
memory.write(
key = VOCAB_KEY,
value = {"vocab": vocab, "mined_at": now(),
"cycle_id": current_cycle_id()},
scope = f"agent:{agent_id}",
ttl = None,
)
audit.log(op="vocab.remined", agent=agent_id,
cycle=current_cycle_id(), terms=len(vocab))
# Downstream scoring reads the post-cycle vocabulary; the cycle id in
# the cache tells downstream code which insight cycle the frame matches.
def score_new_thought(thought, recent_ctx, agent_id):
vocab = memory.recall(query=VOCAB_KEY, scope=f"agent:{agent_id}")
generic = cosine(embed(thought), embed(recent_ctx))
own_axis = sum(w for t, w in vocab["vocab"].items()
if t in tokens(thought))
return 0.5 * generic + 0.5 * own_axis
# Why this works: pure self-corpus-vocabulary on its own cadence can
# drift out of sync with the insight store; pure dream-consolidation
# updates insights without updating the frame used to score them.
# Running both in the same cycle means the agent's frame and its
# durable thinking never disagree about what 'this agent is about'.
# Related: see `dream-consolidation-cycle`, `cluster-capped-insight-store`, `salience-attention-mechanism`.complements → vector-memorycomplements → cluster-capped-insight-storecomplements → salience-attention-mechanismcomplements → dream-consolidation-cycle — Consolidation cycles are a natural place to refresh the vocabulary.complements → semantic-memory