Writing
Agents ·Memory ·Retrieval

How AI agents remember

A field guide to agent memory: the kinds of memory an agent keeps, where each one lives in Postgres, when the agent reads or writes it during a single turn, and when one Postgres stops being enough.

16 min read
Contents

The gist

  • Agent memory splits into three classic kinds borrowed from cognitive science (semantic, episodic, procedural) and four the agent era added (working, entity/graph, reflective, shared).
  • Storage follows the access pattern: exact lookups in a SQL table, semantic recall (including which tools to use) in a pgvector index, relationships in a graph, all inside one Postgres.
  • Most reads are deterministic and run before the model thinks; expensive reads are just-in-time; durable writes happen only when the turn ends.
  • Curate memory by forgetting the trivial and reflecting on hard runs, and treat one Postgres as the default until vector scale, graph depth, or retrieval quality crosses the line into a specialized backend.

An agent with no memory is a brilliant stranger. It reasons well, calls tools, returns an answer, and then forgets everything the moment the turn ends. Memory is the fourth capability, alongside perception, reasoning, and action, and it is the one that turns a clever demo into a system you can rely on across days and sessions. This article is a map of three questions: what an agent stores, where each kind of memory lives in a database, and when the agent reaches for it. It closes on a fourth: when one Postgres stops being enough.

Here is one turn of a memory-aware agent at work. The bright lines are the conversation and its tools; the dim lines beside them are the memory it reads before acting and writes when it stops. The rest of this piece explains those tags.

agent_sessionone turn · reads then writes memory
>triage the spike in checkout 500s and propose a fix
01 / 11
A single turn in an agent terminal. The bright lines are the conversation and its tools; the dim lines with colored tags are the memory it reads before acting and writes when it stops. The rest of this piece is about those tags.

The stateless default

A stateless agent reads its input, reasons over it, and produces output, with nothing carried between calls. That is fine for a one-shot tool and useless for anything real. Without memory an agent cannot run a task that spans many steps, cannot recognize a user who returns tomorrow, and cannot learn from a mistake it made an hour ago. The usual patch, pasting the entire history into every prompt, only delays the failure: the context window fills, latency climbs, cost grows with every turn, and eventually the window overflows.

Memory is the alternative to stuffing. Instead of carrying everything in the prompt, the agent persists what matters outside the context window and retrieves only the slice a given turn needs. The rest of this piece is about how to do that well.

Seven kinds of memory

"Memory" is not one thing. It helps to split it into kinds, because each kind answers a different question and, as we will see, wants a different home in the database.

Three kinds are borrowed straight from how people remember. Semantic memory holds facts: what is true about the user, the domain, and the world. Episodic memory holds experiences: a time-ordered record of what happened and what the agent did. Procedural memory holds skills: how to carry out a task without re-deriving it each time.

Four more kinds were added by the practice of building agents. Working memory is the scratchpad for the task in flight. Entity/graph memory is a map of the things an agent has seen and how they connect. Reflective memory is the lessons an agent draws from its own past runs. Shared memory is a pool that several agents read and write in common.

memory_types3 classic · 4 from the agent era
Semanticfacts
stored

Durable facts about the user, the domain, and the world.

extracted

Stable preferences and attributes pulled from the chat.

in a chatbot

"The user is vegetarian and prefers metric units."

in an agent

A knowledge base the agent retrieves over (RAG).

Episodicexperiences
stored

A time-ordered record of what happened and what was done.

extracted

A short note per turn or event, kept in order.

in a chatbot

"Yesterday you helped me draft an email to Sam."

in an agent

A run log of past executions, for replay and audit.

Proceduralskills
stored

How to carry out a task, plus the catalog of tools.

extracted

Reusable steps and workflows that worked before.

in a chatbot

The system prompt and persona rules.

in an agent

A learned tool sequence (workflow memory) and the toolbox.

Workingtask statenew
stored

The scratchpad for the task currently in flight.

extracted

The current goal, plan, and intermediate results.

in a chatbot

The active conversation in the context window.

in an agent

Plan and partial results held across loop steps.

Entity / graphrelationshipsnew
stored

Entities and the typed edges between them.

extracted

People, systems, and orgs, and how they connect.

in a chatbot

"Sam is your manager; the thread was about Q3."

in an agent

A knowledge graph the agent traverses (Graphiti, Zep).

Reflectivelessonsnew
stored

Lessons the agent draws from its own past runs.

extracted

What went wrong, and the rule that avoids it next time.

in a chatbot

"Ask for the deadline before drafting."

in an agent

A lessons file loaded before similar tasks (Reflexion).

Sharedcommon poolnew
stored

A pool several agents read and write in common.

extracted

Facts and state shared across the team.

in a chatbot

A team-wide FAQ every bot reads from.

in an agent

Multi-agent crew state, a shared blackboard.

Seven kinds, three borrowed from how people remember and four added by building agents. For each: what gets stored, what the agent extracts, and how it shows up in a chatbot versus a broader agentic system.

Shared memory is the multi-agent case

The first six kinds are written by a single agent. Shared memory is the exception: a pool several agents read and write at once. Keep it as an ordinary namespaced table (a scope column every agent filters on), but because there are many writers it is the one place you need concurrency control: row-level locks (select ... for update) or an optimistic version column so two agents do not overwrite each other. The rest of this article assumes a single writer.

These seven kinds are not seven databases. They collapse into a handful of storage shapes, and a single turn touches several of them at once, which is what the rest of this piece shows.

Where memory lives

Here is the engineering core, and it is simpler than the catalog of kinds suggests. The access pattern picks the backend. Ask one question of every read: do I know exactly which rows I want, or only that they should resemble something? The answer routes the data to one of three homes, all of which are just Postgres.

Exact lookups go in a SQL table. Conversational memory is the clearest case. You want this thread's most recent messages, in order. That is an index range scan, not a similarity search.

messages.sql
create table messages (
  id          bigint generated always as identity primary key,
  thread_id   text        not null,
  role        text        not null,          -- 'user' | 'assistant'
  content     text        not null,
  created_at  timestamptz not null default now()
);
 
create index on messages (thread_id, created_at desc);

Reading the recent turns is a single indexed query, no embeddings involved:

recent.sql
select role, content
from messages
where thread_id = $1
order by created_at desc
limit 20;

A tool log or event log has the same shape: rows you fetch by id and time, for replay and audit.

Semantic recall goes in a vector index. When you only know the meaning you want, store an embedding of the content and search by distance. The pgvector extension adds a vector column type and distance operators to plain Postgres.

knowledge.sql
create extension if not exists vector;
 
create table knowledge (
  id        bigint generated always as identity primary key,
  content   text        not null,
  embedding vector(768) not null,
  metadata  jsonb       not null default '{}'
);
 
-- An approximate-nearest-neighbor index (HNSW) over cosine distance.
create index on knowledge using hnsw (embedding vector_cosine_ops);

Retrieval pulls the top-K rows closest to the query embedding. The <=> operator is cosine distance:

recall.sql
select content, metadata
from knowledge
order by embedding <=> $1   -- $1 is the query embedding
limit 5;

Because it is still SQL, you can scope the search with an ordinary filter on the JSONB metadata in the same query:

filtered.sql
select content
from knowledge
where metadata @> '{"source": "papers"}'
order by embedding <=> $1
limit 5;

Knowledge, learned workflows, the tool catalog, entity descriptions, and reflective lessons all share this layout: a content column, an embedding, and some metadata.

One caveat on that layout: vector(768) pins a specific embedding model. Change the model and the dimension and the stored vectors change with it, so version the embedding (a model column, or a parallel index) and plan a backfill rather than migrating in place.

Relationships go in a graph. Entity memory is not really about similarity, it is about connection. Model entities and typed edges as two tables, then walk the graph with a recursive query.

graph.sql
create table entities (
  id        bigint generated always as identity primary key,
  name      text not null,
  kind      text,
  embedding vector(768)          -- so an entity can also be found by meaning
);
 
create table edges (
  src bigint references entities(id),
  rel text   not null,
  dst bigint references entities(id)
);

Pulling the two-hop neighborhood around an entity is a recursive CTE:

neighborhood.sql
with recursive nbrhood as (
  select id, name, 0 as depth
  from entities
  where name = $1
  union all
  select e.dst, t.name, n.depth + 1
  from nbrhood n
  join edges e    on e.src = n.id
  join entities t on t.id  = e.dst
  where n.depth < 2
)
select distinct name, depth from nbrhood order by depth;

Working memory is the exception. The three homes above hold memory that outlives a turn. Working memory, the scratchpad for the task in flight, does not: it is the run's own state, the plan and partial results the loop carries between steps. It lives in the orchestrator, not the long-term stores. You persist it as a checkpoint (a LangGraph PostgresSaver, for instance) so a paused or crashed run resumes where it stopped, not so you can recall it next week.

Tools are memory, too. As an agent gains tools, sending every tool's schema to the model on every call is the same mistake as stuffing the prompt: it is noise the model has to read past. Treat the tool catalog as procedural memory. Store each tool's description as an embedding, exactly like the knowledge table, and retrieve only the few tools whose description is closest to the task at hand.

semantic_tool_retrievalintent → top 3 tools
task · Book me a flight and a hotel for Berlin next month.
book_flightkept
find_hotelkept
convert_currencykept
send_email
run_sql
search_papers
summarize_doc
create_ticket
tools available8
schemas sent to model3 / 8
Tools are procedural memory: store each tool's description as an embedding, then retrieve only the few that match the task instead of sending every schema to the model. The catalog can grow without bloating the prompt.

The retrieval is the same <=> nearest-neighbor search you already run for knowledge; only the rows differ. A catalog of two hundred tools costs the model nothing if the agent loads the three it needs, and the model reasons better when it is not handed a wall of schemas it will never call. This only pays off past a few dozen tools, though. Under roughly thirty, send them all: retrieval adds latency and a failure mode for no real benefit.

One database, not three

Notice that nothing above leaves Postgres. The vector index is an extension, the graph is two ordinary tables, and the conversation log is a plain table with a B-tree. You do not need a dedicated vector store or a separate graph engine to ship a memory-aware agent; you need one Postgres and a clear idea of which question each table answers.

It is tempting to embed everything and search by similarity for all of it. Resist that. Asking a vector index for "this thread's last ten messages, in order" is both slow and wrong: it returns rows that are merely similar, not the ones you asked for. Exact, ordered reads are a B-tree's job. Similarity is the vector index's job. Connection is the graph's job. Match the backend to the question and most of the hard decisions disappear.

When the agent remembers

Knowing where memory lives is half the design. The other half is timing, and it is where most agents go wrong. A single turn touches memory at three distinct moments, and keeping them separate is what makes an agent both cheap and debuggable.

First, deterministic reads, before the model thinks. An agent cannot ask for what it does not know exists, so the core stores load on every turn by a fixed rule, not by the model's choice: recent conversation by thread, relevant knowledge by similarity, similar past workflows, and the entities named in the query. This is the context the model starts from.

Deterministic by default, adaptive when recall matters

Loading by a fixed rule is a deliberate trade. It is predictable and easy to debug, but it leaves recall on the table: the model can often phrase a query better than the user did. The alternatives let the model shape retrieval, query rewriting, HyDE, multi-query, or plan-then-retrieve, at the cost of determinism and extra calls. Start deterministic, and reach for adaptive retrieval on the queries where recall actually matters.

Second, just-in-time reads, inside the loop. Expensive or conditional fetches wait until the model actually asks. A web search fires only when local memory comes up thin, and a full document loads only when the model decides it needs the detail. A lean prompt reasons better than a stuffed one, so the agent pulls detail just in time rather than just in case.

Third, durable writes, at the stop condition. When the turn produces a final answer, the agent persists what it learned: the exchange, the workflow it followed, any new entities, and a reflective lesson. Writing only at the end keeps half-finished reasoning out of long-term memory.

The system below traces one full turn through all three phases. Scroll it: the deterministic reads load from each store, the loop reaches outside only when it must, and the durable writes flow back when the turn ends.

the system

One agent, one Postgres, three stores

Everything the agent remembers lives in three shapes inside a single database: a SQL table for exact, ordered reads, a vector index for meaning, and a graph for connection. Here is the whole system at rest. Scroll to watch one turn move through it.

step 1 · before

Deterministic reads load the context

Before the model thinks, core memory loads by a fixed rule, not by the model's choice: recent conversation from the SQL table, relevant knowledge from the vector index, named entities from the graph. This is the context the turn starts from.

step 2 · the loop

The model reasons, and reaches out just in time

Inside the loop the model reasons over that context. Only when local memory comes up thin does it reach outside, a web search or a document fetch, just in time rather than just in case. A lean prompt reasons better than a stuffed one.

step 3 · after

Durable writes, only when the turn ends

When the turn produces a final answer, the agent writes what it learned back to the stores: the exchange, the workflow it followed, any new entities, and a reflective lesson. Writing only at the stop condition keeps half-finished reasoning out of long-term memory.

tools are memory

The tools live in the vector index too

The catalog of tools is procedural memory. Each tool's description is an embedding in the same vector index, so the agent retrieves the few tools that match the task instead of sending every schema to the model. The catalog can grow without bloating the prompt.

agent_memory
read→loop→write
webagentassemble · reason · actone PostgresSQL tableconversation · tool logvector indexknowledge · workflowgraphentities · relations
the system

In code, the three phases are the top, middle, and bottom of one function. The model chooses only what happens inside the loop; everything around it is fixed.

run_turn.py
def run_turn(query, thread_id):
    # 1. Deterministic reads: assemble context before the model thinks.
    context = assemble(
        recent    = sql_recent_messages(thread_id),       # exact, by thread
        knowledge = knn("knowledge", embed(query), k=5),  # semantic top-K
        workflow  = knn("workflow",  embed(query), k=3),
        entities  = graph_neighborhood(query),            # graph traversal
    )
    write_message(thread_id, "user", query)               # durable, every turn
 
    # 2. The loop: reason, act, fetch just in time.
    messages = [system_prompt(), user(context, query)]
    for _ in range(MAX_STEPS):
        step = model(messages)
        if step.is_final:
            break
        result = call_tool(step)        # e.g. expand_summary(id), web_search(q)
        messages += [step, tool_result(result)]
 
    # 3. Stop condition: persist what the turn learned.
    write_message(thread_id, "assistant", step.text)
    upsert_workflow(query, steps_taken=messages)
    upsert_entities(extract_entities(step.text))
    if turn_was_hard(step):                               # not every turn: reflection costs a model call
        write_lesson(reflect(query, step.text))           # reflective memory, can run async
    return step.text

Read the structure, not the syntax: reads at the top, a tight loop in the middle, writes at the bottom. Because the reads and writes are deterministic, every run starts and ends the same way, which is exactly what makes a misbehaving agent possible to debug.

One line in there hides real work: assemble. Four stores can return more than fits the prompt, so assembling context is a budget problem, not a concatenation. Give each store a token quota, dedupe across them (the same fact often surfaces from both knowledge and the graph), and set a priority for conflicts: an explicit user statement outranks an inferred one, and fresh outranks stale. The model only ever sees what survives that pass.

Keeping memory from rotting

A memory that only grows eventually buries the agent in its own history. So writing needs discipline, the same way human memory keeps what matters and lets the trivial fade.

Forgetting. Not every fact earns a permanent place. A durable preference is worth a write; a one-off aside is not. Deciding what to keep takes judgment, so it is one of the few memory operations that belongs to the model rather than to a rule.

Forgetting is a feature

The instinct is to save everything, just in case. But noise retrieved is noise reasoned over. An agent that writes selectively keeps its future retrievals sharp; an agent that hoards slowly poisons its own context.

Consolidation. Forgetting prunes; consolidation compresses. An episodic log that only grows will eventually dwarf its own useful signal, and you cannot delete your way out of that. So fold old episodes upward: summarize a run into a paragraph, a week of runs into a page, the way human memory keeps the gist and lets the transcript fade. Tiered schemes like this, recent detail in full and older memory as recursive summaries, are what let a long-lived agent stay both informed and small.

Reflection. After a hard task, the agent writes down what it learned. The next time a similar task begins, that lesson loads with the rest of memory, and the agent avoids repeating the mistake. This is the loop that lets an agent get better at a job over time instead of starting fresh each morning. Reflection is a model call, though, so it does not belong on every turn. Run it only when a turn was hard, failed, or surfaced something new, and run it out of band when you can, so it never sits on the response path the user is waiting for.

When one Postgres is not enough

The design so far is deliberately monolithic: one Postgres, three access patterns. It will carry you much further than the urge to add infrastructure suggests. But it is not the end state for every system. Three axes each have a line where a specialized backend stops being premature and starts being correct. Be honest about where yours sit.

when_to_graduatepostgres → first lever → graduate
1stay on postgresPostgres + pgvector (HNSW)

Up to ~1M vectors, modest QPS

Transactional rows and vector search live in one database.

2first leverTune HNSW: m, ef_construction, ef_search

Recall or latency slips first

Defaults are conservative; tuning buys a lot before any migration.

3graduateDedicated vector DB: Qdrant, Weaviate, Milvus

Past ~10M vectors or high QPS

Quantization, sharding, and filtered search that stay fast at scale.

pgvector comfortup to ~1M
graduate near10M+ or high QPS
One Postgres is the right default, and it goes further than most expect. Each axis has a line where a specialized backend earns its keep. Switch axes to see where that line is and what to reach for. Migrate an axis only when you measure its line, not before.

On vector scale, pgvector with a tuned HNSW index is comfortable into the low millions of rows. Past that, or under high query load, a dedicated vector database (Qdrant, Weaviate, Milvus) pays for itself: quantization, sharding, and filtered search that stay fast where Postgres starts to strain.

On graph depth, two tables and a recursive CTE are fine for shallow, occasional traversals. They break when edges need weights or timestamps, when traversals run deep and hot, or when you want graph algorithms like community detection or centrality that SQL will not express well. That is where a property-graph database (Neo4j, Memgraph) and Cypher earn their place, and where a temporal knowledge-graph layer (Graphiti, Zep) keeps facts that were only true for a while. In relationship-heavy, time-aware work, this is usually the first axis to graduate.

On retrieval quality, cosine similarity alone is a simplification. Production retrieval is typically hybrid: lexical search (BM25) fused with dense vectors to catch the exact terms embeddings miss, then a cross-encoder reranker over the top candidates. Pair that with adaptive retrieval, query rewriting, HyDE, or multi-query, when a single pass leaves recall on the table.

Start simple, graduate on evidence

None of this argues against starting on one Postgres. It argues against guessing. Build the simple design, measure where it strains, and graduate the one axis that crosses its line, not all three on a hunch. Premature specialization costs more than it saves.

The takeaway

Three questions carry the whole design. The taxonomy answers what to store: seven kinds, three of them borrowed from how people remember and four added by building agents. Postgres answers where it goes: exact reads in a SQL table, semantic reads (knowledge, and the tools to act on it) in a pgvector index, relationships in a graph, all in one database. The loop answers when to touch it: deterministic reads before the model thinks, just-in-time reads while it works, durable writes when it stops. A fourth question waits at scale: one Postgres is the right default, and you graduate a single axis, vector scale, graph depth, or retrieval quality, only once you have measured it past the line. Get those right and you have crossed most of the distance between a demo that forgets you and a system that does not.

All writing