AI memory is a writing problem, not a search problem

The moment you say "an AI that remembers you," someone in the room says "oh, so RAG." Embed every past message, retrieve the most similar ones, paste them into the prompt. Done.

I built an AI companion app where characters are supposed to remember you across hundreds of messages, over weeks. I tried the RAG version first. It is not what makes memory work, and the distance between "retrieve similar old messages" and "actually remembers you" turned out to be the entire problem.

Here is the reframe that fixed it: memory is a writing problem, not a search problem.

Search assumes the answer is already written down

RAG is great when the truth already lives in a document you have: a manual, a wiki, a knowledge base. You are not creating knowledge, you are finding the page it is on.

A conversation is not like that. Nothing is written down yet. The raw transcript is mostly noise: "lol," "ok wait," half-finished thoughts, and the character's own replies. Embed all of it and pull the top few "similar" lines and you get a pile of out-of-context fragments, not an understanding of a person. You can feel it in the output. The character quotes something from three days ago in the wrong tone, or latches onto a passing joke because it happened to be semantically close to the new message.

The fix is to stop storing the transcript and start taking notes.

Memory is what you choose to write down

After each turn, before anything is "remembered," a small cheap model does one job: write a single sentence about what just happened. Not the message. A note. "User got laid off last week and is trying to act fine about it." One line, present tense, about the user. Plus a couple of tags: who was named, whether it was a real statement or a joke, the emotion underneath.

That tiny act of compression is the whole game. Every turn, you decide what is worth keeping and in what shape. A joke gets tagged as a joke so it never gets treated as a fact. A throwaway "ok" gets skipped entirely. What lands in memory is already clean, already interpreted, already small.

This is the part RAG skips. RAG has no write path. It indexes whatever you throw at it and hopes similarity sorts it out later. Taking notes moves the judgment to the front, where it is cheap, instead of the back, where it is impossible.

The thing that actually makes it feel like it knows you

Notes are events. Knowing someone is more than a list of events. So on top of the running notes there is a second, slower layer: a maintained profile. A short card. Who they are, how they talk, the people in their life, the threads still open.

It is rewritten periodically, not appended to. And it follows sticky rules, which are pure product decisions, not infrastructure:

• Identity (name, job, where they live) stays until the user contradicts it.
• Named people stay until the user says they are gone.
• Major life events (a breakup, a layoff, a death, an engagement) stay permanently. Those are not trivia. They are who the person is now.
• Voice and smaller threads are allowed to rotate as the conversation moves.

The key word is "rewritten." If you only append, the profile bloats forever and the character starts sounding like it is reading a dossier about you. Rewriting against a hard size cap forces it to stay a portrait, not a transcript. After thirty rewrites it is still about eighty words. Bounded on purpose.

That small curated card is what carries the feeling of being known. Not the vector search.

The trap nobody warns you about: the model poisons its own memory

This is the one that cost me the most, and it stays invisible until it quietly ruins a conversation.

If you embed the raw history, you embed the character's replies too. Now the model's own output becomes "memory." A turn where it glitched and said something off, or invented a detail to fill a gap, gets stored with the same weight as something the user actually said. A few hundred turns later the character "remembers" things that never happened, because it is reading its own hallucinations back to itself. Once it starts, it compounds.

Search-based memory has this bug baked in. The fix lives on the write side again: a note is about the user's turn, never the character's answer. If the user only asked a question, you record the question, not the reply. If a past exchange came out garbled, you leave it out of the summary entirely. You are firewalling the model from its own output. RAG over the transcript has nowhere to put that firewall.

Retrieval is the small part, and most turns skip it

There is still a vector search in here. It just is not the system. It is a supporting actor that mostly stays on the bench.

In a fresh conversation it does nothing, because there is nothing old to fetch. It only switches on once there are enough notes to be worth searching, and even then it returns a handful, sitting alongside three sources the model trusts more: the profile card, a rolling summary, and the last few notes verbatim. On a typical turn, "what it remembers" is the profile plus the recent notes, with zero search and zero extra model calls at read time.

When I measured it, the version that wrote good notes and kept a tight profile was both better and cheaper than the version that embedded everything and retrieved hard. Fewer calls, smaller context, no drift over a thousand turns. Frugal and sharp turned out to be the same habit here too.

So, is it RAG?

There is an embedding model and a vector index in the stack, so some checkbox technically says yes. But calling it RAG misses the point so completely that it is the wrong word.

RAG answers "where is the page with the answer." A companion's memory answers "who is this person, what have they told me, and what should I never forget." The first is search. The second is note-taking, curation, and a few firm rules about what gets to persist and what the model is not allowed to remember about itself.

If you are building anything that has to remember a person over time, do not start with a vector database. Start with the question every good assistant, human or otherwise, answers without thinking: what is worth writing down, and what should I quietly let go.