Giving an AI long-term memory with vectors and RAG

Ask most chatbots about something you told them last week and they have no idea. Every conversation starts from zero.

For real work this is a deal breaker. Imagine a company that talks to 40 clients over email, WhatsApp and shared files. A useful AI should answer questions like "what did this client say about the deadline?" or "what is still pending for this account?". That needs memory.

The standard way to build this is called RAG: Retrieval-Augmented Generation. The name sounds hard. The idea is simple: before the model answers, find the right notes and give them to it.

#The big picture

#Step 1: match every message to the right client

This step is easy to skip and very important. Memory is useless if it is mixed up. A message about client A must never show up in an answer about client B.

So before anything is stored, each message is matched to an account. I use simple signals first: the email domain, the phone number, the people in the thread. Only when those fail does a model guess, and a low-confidence guess goes to a person to confirm.

#Step 2: cut long text into chunks

A long email thread is too big to use as one piece. So I cut it into chunks of a few paragraphs each, with a small overlap so a sentence is never cut in half without context.

Each chunk keeps its metadata: which client, which channel, who sent it, and when.

#Step 3: turn each chunk into a vector

An embedding model turns text into a list of numbers, called a vector. The magic is that texts with similar meaning get similar numbers. "The launch is moved to Friday" and "we are pushing the release to end of week" end up close together, even though they share almost no words.

#Step 4: answer from memory

When someone asks a question, the same thing happens in reverse:

The important part is "same client only". The search is filtered by account before it looks at meaning. This one filter prevents the worst kind of mistake: leaking one client's information into another client's answer.

const chunks = await memory.search({
  vector: await embed(question),
  filter: { clientId },   // never search across clients
  limit: 8,
});

const answer = await model.generate({
  system: "Answer only from the notes. If the notes do not say, say you do not know.",
  prompt: `${question}\n\nNotes:\n${chunks.map((c) => c.text).join("\n---\n")}`,
});

#Memory for conversations, not just documents

A conversational AI needs a different kind of memory too. It should remember things about the person: what they shared, the tone they like, what is going on in their life.

For that, I do not store every message forever. After a conversation, a model writes a short summary of what is worth remembering. Next time, those summaries are loaded first. It is closer to how people remember: not every word, just what mattered.

#Mistakes I made so you do not have to

• Chunks too big. The search found the right email but the answer was buried in noise. Smaller chunks gave better answers.
• No metadata. Without dates, the AI happily quoted an old deadline that had already changed. Now newer chunks win when two disagree.
• No "I do not know". Without that rule in the prompt, the model fills gaps with guesses. Tell it, clearly, that not knowing is allowed.
• No sources. Every answer should show which messages it came from, so a person can check in one click.

#The short version

Memory is not a feature of the model. It is a system you build around it: match, chunk, embed, store, filter, retrieve. Get those steps right and even a small model feels like it really knows your work.