AI agents · OpenClaw · self-hosting · automation

Quick Answer

Knowledge Layer Architecture: Storage, Retrieval, Updates

Published:

The short answer

A knowledge layer has four parts: ingestion, storage, retrieval and freshness. Design each around one rule: every answer must trace back to a specific version of a specific source that the asking user is allowed to see. This page is the architecture spoke of our guide to choosing an AI knowledge layer.

LayerDesign decisionDefault that works
IngestionParse, chunk, attach metadataStructure-aware chunks of 300–800 tokens; keep headings and page numbers
StorageWhere text, vectors and metadata livePostgreSQL + pgvector until scale forces a dedicated engine
RetrievalHow candidates are found and rankedHybrid search → permission filter → rerank top 30–50 to 5–10
FreshnessWhen each source is updatedEvent-driven for operational data; scheduled incremental for documents
AuditWhat was retrieved per answerLog query, chunk IDs, versions and scores for every answer

Storage design

Store each chunk as one record with:

  • Text and its embedding (record the embedding model name and version — you will change it).
  • Provenance: source URL or path, document ID, version or hash, owner, ingestion time.
  • Access control: the groups or tenant that may read it, copied from the source system.
  • Status: active or superseded, so old versions stop competing in retrieval without being lost.

Choosing the store. pgvector keeps vectors in PostgreSQL, so metadata filters, permissions and joins use one database and one backup. Move to a dedicated engine when you need native hybrid search at large scale, multi-vector representations or very high query volume: Qdrant (v1.20.2, November 16, 2026), Weaviate (v1.40.6, November 19, 2026) and Milvus (v3.0.6, November 11, 2026) all support hybrid queries. Compare them in best vector databases.

Retrieval design

  1. Hybrid search. Combine dense vectors (meaning) with keyword or sparse search (exact terms: product codes, error messages, names). Pure vector search misses exact identifiers.
  2. Filter by permissions at query time, using the requesting user’s identity. Filtering at indexing time breaks as soon as someone’s access changes.
  3. Retrieve wide, then rerank. Pull 30–50 candidates and let a reranker choose the final 5–10; see best reranker models.
  4. Prefer the newest active version when two chunks say different things, and pass the version date to the model so it can say “as of”.
  5. Return citations: chunk IDs and source links travel with the answer.

Update frequency by source

Set freshness per source, from how fast it changes and what a stale answer costs:

SourceChange rateUpdate methodTarget lag
Support tickets, CRM records, incidentsConstantChange events or webhooks (CDC)Minutes
Product documentation, runbooksDaily–weeklyIncremental sync on modified-sinceHourly to daily
Internal wiki, shared drivesDailyIncremental syncDaily
Policies, contracts, manualsMonthly or on releaseTriggered on publish; supersede old versionSame day as release
Code repositoriesPer commitRe-index changed files on mergePer merge

Re-embed only what changed (compare content hashes). Schedule a full rebuild only when you change the embedding model or chunking strategy, and run it side by side before switching traffic.

Prove it works before you scale

Collect 50–100 real questions with known answers and re-run them after every chunking, embedding, reranker or freshness change. Track retrieval hit rate separately from answer quality, so you can tell a retrieval failure from a model failure. Chunking choices are compared in best chunking libraries for RAG; the step-by-step build order is in how to build an agent knowledge layer.

Last verified: November 20, 2026.

Sources