Local Agent Memory — Retrieval Architecture for CLI and VS Code

01 · Decision

Use a thin skill over a local retrieval service.

The skill should contain the operating procedure. The knowledge base should remain outside the prompt. Retrieval indexes should be rebuildable. Every answer should be traceable to exact source evidence.

Authoritative

Files

Readable, reviewable, diffable, portable, and suitable for normal repository governance.

Deterministic

SQLite

Tracks authority, scope, versions, symbols, relationships, lexical search, and session state.

Semantic

LanceDB

Provides embedded vector retrieval without requiring a separate local service.

Precision

Reranker

Scores the query against candidate passages before any evidence enters the agent context.

02 · Knowledge model

Separate truth, indexes, and active context.

The system should maintain three different states. Mixing them creates stale prompts, opaque memory, and unsafe writeback.

1

Canonical knowledge

Repository files, documentation, source code, ADRs, RFCs, tests, procedures, and reviewed notes.

2

Derived projections

Document cards, section summaries, claims, aliases, embeddings, lexical indexes, and relationships.

3

Evidence packages

Small, query-specific sets of ranked excerpts with source paths, line ranges, authority, and conflicts.

4

Session working set

Task state, selected evidence, open questions, and loaded-content hashes. Ephemeral by default.

Why this separation matters

Rebuildable

Indexes can be discarded

An embedding-model change should trigger re-indexing, not a migration of canonical knowledge.

Auditable

Evidence stays inspectable

A generated summary never replaces the source passage that supports it.

Bounded

Context remains temporary

The active working set can be evicted without deleting durable knowledge.

Avoid the “single vector database” shortcut A vector store is useful for candidate recall. It is not a sufficient authority model, version graph, audit log, session store, or source-of-truth repository.

03 · Progressive disclosure

Move from intent to evidence in seven controlled steps.

The agent should start with almost no knowledge-base content. Each expansion should be justified by the task and should reveal only the next useful layer.

Level 0

Skill discoveryName and description only

≈ 50–120 tokens

Level 1

Retrieval procedureConcise SKILL.md workflow

≈ 800–1,500 tokens

Level 2

Domain routingCandidate knowledge areas

≈ 150–350 tokens

Level 3

Evidence cardsTitles, why selected, short excerpts

≈ 500–1,200 tokens

Level 4

Exact sourceStrongest sections and line ranges

≈ 800–2,000 tokens

Level 5

Relationship expansionADRs, tests, superseded designs

On demand

Level 6

Forked investigationDedicated subagent or deep analysis context

Exceptional

Disclosure policy

Reveal

Evidence that directly supports the task.
Current and authoritative sources first.
Conflicts that could change the answer.
Exact source text after a candidate is selected.
Related decisions only when rationale is required.

Suppress

Previously loaded and unchanged passages.
Neighboring content with no decision value.
Generated summaries lacking source references.
Superseded guidance unless history is requested.
Low-confidence results presented as authority.

04 · Recall pipeline

Retrieve broadly. Rank narrowly. Expand exactly.

High accuracy comes from combining retrieval channels, applying hard filters early, reranking with a stronger model, and using source authority as a first-class signal.

01

Classify retrieval intent

Determine whether the task is an exact lookup, procedure, troubleshooting question, architecture rationale, policy interpretation, historical decision, or cross-document synthesis.

02

Apply hard filters

Constrain repository, branch, component, environment, validity period, status, source type, language, and access scope before semantic recall.

03

Run parallel retrieval

Search exact identifiers, symbols, paths, SQLite FTS5/BM25, dense vectors, aliases, and explicit relationships independently.

04

Fuse candidate ranks

Use reciprocal-rank fusion rather than adding incompatible lexical and vector scores.

05

Apply authority and freshness rules

Boost accepted decisions, current implementations, exact symbol matches, and validated procedures. Penalize deprecated, stale, or generated-only material.

06

Rerank locally

Use a cross-encoder to score the query against the top candidate passages. This is the final precision gate before context injection.

07

Expand source evidence

Retrieve exact line ranges, parent sections, linked ADRs, tests, or procedures only for the selected candidates.

08

Assemble a bounded evidence pack

Return query interpretation, scopes, top evidence, conflicts, coverage, and available next expansions.

Rank fusion

Why not add scores?

BM25 scores, cosine similarity, and model reranker scores live on different scales. Direct addition makes ranking unstable and hard to interpret.

Recommended default

Reciprocal-rank fusion combines list position rather than raw score. Introduce learned or weighted fusion only after collecting a labeled evaluation set.

RRF scoring

RRF(document) = Σ [ weight(retriever) / (k + rank(document, retriever)) ]

initial settings:
  lexical candidates: 50
  semantic candidates: 50
  exact/symbol candidates: 20
  relationship candidates: 20
  rerank fused top: 30–100
  final evidence sections: 2–5

05 · Storage design

Give each store one clear responsibility.

The design remains understandable because the canonical layer, control plane, semantic projection, and local model runtime have distinct duties.

Component	Use for	Do not use as	Operational property
Repository files	Canonical knowledge, policy, code, ADRs, runbooks, examples	Fast global retrieval index	Reviewable and portable
SQLite + FTS5	Metadata, authority, versions, relationships, exact and lexical retrieval, session state	Primary dense ANN engine	Embedded and transactional
LanceDB	Document, section, chunk, and symbol embeddings	Canonical governance record	Embedded semantic recall
Local embedding model	Query and corpus vectors	Final passage judgment	Offline and replaceable
Local reranker	Query-to-passage precision scoring	Broad first-stage recall	Higher cost, smaller candidate set
Session ledger	Loaded evidence, hashes, context state, evictions	Durable organizational knowledge	Ephemeral and query-specific

Index at multiple structural levels

Coarse

Document cards

Title, abstract, authority, scope, major topics, and document status. Useful for routing.

Focused

Section cards

Heading path, purpose, summary, entities, and parent document. Useful for main retrieval.

Exact

Evidence chunks

Exact source text, line range, parent section, content hash, and source version. Useful for final context.

Use structure before token windows Parse Markdown headings, code symbols, ADR sections, tables, procedures, and API definitions. Fall back to token-based splitting only when a structural unit is too large.

06 · CLI and skill contract

Make the skill a policy layer and the CLI a deterministic tool.

The skill explains when and how to retrieve. The CLI owns indexing, filtering, search, expansion, tracing, and evaluation. It emits structured JSON for reliable agent consumption.

Recommended repository layout

Repository layout

repository/
├── skills/knowledge-recall/
│   ├── SKILL.md
│   ├── references/
│   └── scripts/kb.ts
├── cli/src/
│   ├── commands/
│   └── retrieval/
├── knowledge/
│   ├── sources/
│   ├── maps/
│   ├── glossary/
│   └── generated/
├── .knowledge/
│   ├── catalog.sqlite
│   ├── lance/
│   └── models/
└── evals/
    ├── retrieval-cases.jsonl
    └── expected-evidence.jsonl

Core commands

Command	Purpose
`kb route`	Identify candidate knowledge domains
`kb search`	Run hybrid candidate retrieval
`kb open`	Read exact authoritative evidence
`kb expand`	Traverse parents, siblings, decisions, and tests
`kb trace`	Show provenance, versions, and history
`kb context`	Inspect the active session working set
`kb evaluate`	Run retrieval benchmarks

Skill behavior

Must

Retrieve before claiming

When repository or organizational context is needed, search first and ground the answer in exact evidence.

Must

Expose uncertainty

Return conflicts, missing coverage, stale sources, and abstention signals rather than smoothing them away.

Must not

Persist silently

Agents can propose changes. They should not directly mutate reviewed knowledge without a controlled approval path.

Example agent-facing search response

knowledge-search-response.json

{
  "interpretation": {
    "intent": ["architecture-decision"],
    "scopes": { "repository": ["core-platform-ai"] }
  },
  "results": [
    {
      "id": "section:remote-mcp-identity:local-proxy",
      "title": "Local identity proxy",
      "sourcePath": "knowledge/sources/mcp/identity-architecture.md",
      "lineRange": [142, 191],
      "whySelected": "Defines the local proxy and server validation boundary",
      "authority": "accepted-architecture",
      "status": "current",
      "retrievalChannels": ["fts5", "vector", "relationship"],
      "alreadyLoaded": false
    }
  ],
  "conflicts": [],
  "coverage": 0.91,
  "abstain": false
}

07 · Context ledger

Track what the agent has already seen.

Repeatedly loading the same architecture sections is a hidden context tax. A session ledger converts retrieved evidence into a managed working set.

NewEvidence not previously loaded in the session

ChangedSame stable ID, different content hash or version

LoadedAlready present and unchanged; suppress by default

SupersededHistorically relevant but not current authority

ConflictingCompeting sources that may change the conclusion

EvictedRemoved from active context but still retrievable

Context ledger schema

TypeScript

interface ContextLedgerEntry {
  sessionId: string;
  evidenceId: string;
  contentHash: string;
  loadedAt: string;
  tokenCount: number;
  reason: string;
  relevance: number;
  state: "active" | "summarized" | "evicted";
}

Highest-leverage context optimization Before returning evidence, compare stable IDs and content hashes against the session ledger. Return only new, changed, or explicitly requested material.

08 · Governance and safety

Read by default. Propose changes. Review before persistence.

Persistent developer memory can carry prompt injection, stale instructions, secrets, and unsupported conclusions across sessions. Write controls are therefore part of retrieval quality—not a separate concern.

Allowed

Agent proposals

Propose a corrected fact with source evidence.
Draft an ADR or runbook update.
Add an alias or code-symbol mapping.
Mark a source as potentially superseded.
Record a candidate troubleshooting observation.

Blocked

Silent mutation

Promoting arbitrary retrieved text into instructions.
Overwriting reviewed architecture from a chat summary.
Persisting task-local assumptions as durable facts.
Indexing secrets or credentials.
Deleting evidence because it conflicts with a preferred answer.

Authority precedence

Precedence model

security and organizational policy
  > repository-controlled instructions
  > directory or component instructions
  > task-specific plan
  > explicit user request
  > approved personal preference
  > auto-generated memory
  > model inference

How should knowledge changes be submitted?

Use a command such as kb propose --type correction --evidence .... The command should generate a patch, issue, or candidate record that includes the source evidence, proposed change, author, model provenance when applicable, and review status.

What should never become an instruction?

Untrusted documents, tool outputs, retrieved webpages, generated summaries, and prior agent messages should remain data. Only explicitly governed files or policies should participate in instruction authority.

How are secrets handled?

Scan content before indexing and before returning it. Store exclusion rules by path and classification. Record redaction events. Do not depend on prompt instructions to protect credentials.

09 · Evaluation

Optimize evidence utility, not similarity.

A technically impressive vector index can still return stale, weak, or redundant evidence. Evaluation must incorporate authority, freshness, citation support, context cost, and abstention.

Recall@10Did the system retrieve the needed evidence?

MRRHow high was the first correct result?

nDCG@10Were the strongest sources ranked first?

Stale rateHow often did superseded guidance surface?

Conflict recallWere material contradictions exposed?

Citation accuracyDid the exact source support the claim?

Incremental ratioHow much returned context was truly new?

AbstentionDid the system decline unsupported answers?

Primary optimization target Maximize correct, authoritative, usable evidence per context token and unit of retrieval latency.

Utility objective

utility =
  correct × authoritative × usable evidence
  ─────────────────────────────────────────
           context tokens × latency

10 · Implementation roadmap

Start with deterministic retrieval. Add sophistication only when evaluation justifies it.

The architecture is intentionally incremental. Each phase produces a usable local skill and creates measurements for the next decision.

Reliable baseline

Canonical files and document parser
SQLite schema and FTS5
route, search, open
Stable IDs and source hashes
Initial real-task evaluation set

Semantic precision

LanceDB section and chunk vectors
RRF hybrid fusion
Local reranker
Authority and freshness weighting
Conflict detection

Context intelligence

Session context ledger
Relationship expansion
Write proposals and review flow
Forked investigations
Optional remote service adapter

Definition of done

Functional

CLI returns structured evidence with exact provenance.
Skill can answer common engineering questions without loading the full knowledge base.
Changed and previously loaded evidence are distinguishable.
Unsupported questions trigger abstention or clarification.

Operational

Indexes rebuild from canonical sources.
Model versions and embedding dimensions are recorded.
Secrets and excluded paths are not indexed.
Evaluation runs are repeatable and versioned.

11 · Sources and provenance

Primary references behind the design.

The architectural recommendation is an analysis synthesized from these primary specifications, documentation sets, and research papers. Product-specific capabilities should be revalidated before implementation because local-agent tooling changes quickly.

01

Agent Skills specificationProgressive loading of skill metadata, instructions, scripts, and references.

Open source

02

VS Code agent skillsSkill discovery, custom locations, and agent customization behavior.

Open source

03

SQLite FTS5Embedded full-text search, tokenization, prefix indexing, and BM25 ranking.

Open source

04

LanceDB documentationEmbedded vector storage, TypeScript support, filtering, full-text, hybrid search, and reranking.

Open source

05

Anthropic contextual retrievalEvidence for contextualized chunks, lexical-plus-vector retrieval, and reranking.

Open source

06

Qdrant hybrid queriesReciprocal-rank fusion and multistage hybrid retrieval concepts.

Open source

07

Lost in the MiddleResearch showing that long context availability does not ensure reliable use of evidence at every position.

Open source

08

Qwen3 embedding and reranking modelsCandidate local model family for multilingual, code, embedding, and reranking workloads.

Open source

09

EmbeddingGemmaSmaller candidate embedding model for resource-constrained local environments.

Open source

10

sqlite-vec project statusRelevant when evaluating a single-SQLite implementation; pre-1.0 maturity should be considered.

Open source

Evidence boundary The sources establish product capabilities and retrieval research. The final stack selection, precedence model, session ledger, and phased rollout are architectural recommendations derived for the stated local CLI and VS Code use case.

Store knowledge locally. Reveal only what the agent needs.

Use a thin skill over a local retrieval service.

Files

SQLite

LanceDB

Reranker

Separate truth, indexes, and active context.

Why this separation matters

Indexes can be discarded

Evidence stays inspectable

Context remains temporary

Move from intent to evidence in seven controlled steps.

Disclosure policy

Retrieve broadly. Rank narrowly. Expand exactly.

Rank fusion

Why not add scores?

Recommended default

Give each store one clear responsibility.

Index at multiple structural levels

Document cards

Section cards

Evidence chunks

Make the skill a policy layer and the CLI a deterministic tool.

Recommended repository layout

Core commands

Skill behavior

Retrieve before claiming

Expose uncertainty

Persist silently

Example agent-facing search response

Track what the agent has already seen.

Context ledger schema

Read by default. Propose changes. Review before persistence.

Agent proposals

Silent mutation

Authority precedence

Optimize evidence utility, not similarity.

Start with deterministic retrieval. Add sophistication only when evaluation justifies it.

Reliable baseline

Semantic precision

Context intelligence

Definition of done

Functional

Operational

Primary references behind the design.