RAG explained for founders: making an AI assistant answer from your own data

The problem it solves

A language model knows a great deal about the world and nothing about your business. Ask it what your refund policy is and it will produce a confident paragraph that sounds like a refund policy. It will not be yours. Every AI assistant that has to answer from a company's own documentation, tickets, contracts or records runs into this, and the fix is not to train the model on your data. Training is slow, expensive, and out of date the day it finishes. The fix is to look things up.

Retrieval-augmented generation, RAG for short, is the pattern for that. Instead of asking the model to remember, you find the relevant passages in your own data first and hand them to the model along with the question. The model reads and answers, and it can show where the answer came from.

How it works, in one paragraph

Your documents are split into passages, each passage is turned into a numeric representation that captures its meaning, and those are stored in a database that can find "passages similar to this question" quickly. When a question comes in, the system retrieves the closest passages, builds a prompt that contains them and the question, and asks the model to answer from that material only, citing the passages it used. Think of it as an open-book exam: the model's job is reading and reasoning, not recall.

When you do not need it

Not every assistant needs retrieval. If your entire knowledge base is a few dozen pages, it may simply fit in the model's context window, and putting it all in the prompt is simpler and more accurate than retrieving parts of it. Anthropic's own guidance is to skip RAG when the knowledge base is small enough to fit. Retrieval earns its complexity when the data is large, changes often, or has to be filtered by who is asking.

What good looks like

A RAG assistant that holds up does four things. It answers from your material and shows the source, so a person can check. It says it does not know when the material does not contain the answer, instead of inventing one. It respects permissions, so a user never sees a passage they were not allowed to read. And it stays current, because the passages it searches are the ones in your systems today, not a copy taken last quarter.

Each of those is where a naive build fails.

Where it fails in production

Retrieval misses. A passage split out of its document loses context. "Revenue grew 3%" is meaningless without the company and the quarter, so the search fails to find it when it should. Anthropic measured this and published a fix, contextual retrieval, which prepends a short description of the surrounding document to each passage before indexing; in their tests it cut retrieval failures by about a third on its own, by half when combined with keyword search, and by two-thirds with reranking. The lesson for a founder is that the quality of an assistant is mostly the quality of what it retrieves.

Stale data. Most first builds copy documents into a separate search index. The copy drifts from the source within days, and the assistant confidently answers from last month's price list.

Permission leaks. The same separate index, shared across customers or departments, is how one tenant's data ends up in another tenant's answer. A recent study of production RAG systems names tenant leakage and staleness as two of the three failures that come from splitting the search layer away from the database that knows who is allowed to see what.

No measurement. An assistant that was judged on a demo has no number to defend. When a model changes, or the documents do, nobody knows whether it got worse.

What it takes to build one that holds up

Keep the data in one place. We put the vectors next to the records they describe, in PostgreSQL with vector search, so permissions, freshness and retrieval are one system rather than three that have to be kept in sync. The same study found that this design removed cross-tenant leakage entirely and cut filtered query latency by large margins, and it removes the synchronization code that is the usual source of stale answers. A dedicated vector store earns its place when the scale demands it, not before.

Search two ways. Meaning-based retrieval finds passages that say the same thing in different words; keyword search finds the exact product code or error message. Good systems use both and rerank the results before the model sees them.

Build the evaluation set first. Fifty real questions with the answers a person would accept, run against the assistant before launch and again on every change. It is the only way to know the number moved.

Monitor after launch. What people ask, what the assistant could not find, where it said "I don't know". That log is the roadmap for the next month.

Choose the model per use case. Claude, GPT, Gemini or an open model, on quality, cost and where your data is allowed to go. When the data cannot leave your environment, an open model runs inside it.

Keep it in your accounts. The model provider, the database, the cloud: yours, with the settings that keep your data out of anyone's training.

What to bring to a first conversation

Which documents or records the assistant should answer from, and roughly how many. Who is allowed to see what. Ten real questions your customers or staff actually ask, with the answers you would want. And what a good answer looks like to you, because that becomes the first evaluation set. With that, the shape of the build is clear within a short call, and so is whether an AI feature is the right answer to the problem at all.