← Journal· AI · Aug 14, 2026 · 4 min read

Retrieval augmented generation explained for founders

Almost every founder who asks for an AI feature says they want a model trained on their data. Most of them want retrieval instead, which is cheaper, faster to change, and fails in ways you can actually see.

Almost every founder who asks us for an AI feature describes it the same way. We want it trained on our data. Nine times out of ten, training is not what they want at all. They want retrieval, which costs a fraction as much, can be changed on a Tuesday afternoon without touching the model, and fails in ways you can actually see and fix.

What retrieval augmented generation actually does

RAG is simpler than the acronym suggests. The model never learns your documents. When someone asks a question, you search your own content first, take the handful of passages that look relevant, and hand those passages to the model along with the question. The model answers using what it was given. That is the entire idea. The retrieval half is a search engine. The generation half is a competent writer who has been passed the right page.

Two properties make this the default choice for most products. You update the system by editing a document, not by retraining anything, so your Tuesday policy change is live on Tuesday. And because the answer came from specific passages, you can show them. A user who can click through to the paragraph an answer came from will forgive the occasional miss. A user staring at a confident paragraph with no source will not trust any of it, including the parts that were right.

Fine tuning is usually the wrong answer

Fine tuning teaches a model shape and style. It is good at making output land in a consistent format, adopt a house voice, or follow a labelling scheme you care about. It is poor at teaching facts, because facts change and a fine tuned model is a snapshot with an expiry date on it. If your knowledge shifts every few weeks, and for most companies it does, you would be paying to bake yesterday into the product. When a client insists on fine tuning, we ask what changes more often, the wording of the answers or the answers themselves. That question usually settles it.

The more useful thing to internalise is that the model is not where your project will fail. If your search returns the wrong three paragraphs, the best model on the market will write a fluent, confident, wrong answer from them. Almost all the engineering effort in a RAG build goes into the retrieval side, and almost none of it is glamorous. How documents get split. What metadata travels with each piece. Which version of a file wins. Who is allowed to see what.

Before we agree to build one, here is what we check:

The knowledge exists in writing. If the real answers live in someone's head or in call recordings nobody has transcribed, that is a content project first and an AI project second.
The documents have structure worth keeping. Headings, dates, versions. Splitting a policy PDF into blind five hundred character chunks throws away most of what made it answerable.
Permissions are enforced at retrieval, not in the prompt. If a user is not allowed to read a document, it must never enter their context. Asking a model to keep a secret is not access control.
Each question has one correct source. If three documents contradict each other, retrieval will surface all three and the model will pick one. Fix the content before you build on top of it.
Thirty real questions with expected answers, written down before the build and rerun after every change. Without them, every tweak to your retrieval is a guess.

The failure we see most often is not technical at all. A company points the system at a shared drive that has accumulated eleven years of files, including four versions of the same handbook and a folder of drafts someone meant to delete in 2021. The retrieval works perfectly and returns a passage from the 2021 draft. Nobody can explain why the answer is wrong, because the answer is a faithful reading of a document that should not exist. Cleaning the source material is boring, and it is the highest value week in the whole project.

If your search returns the wrong three paragraphs, no model will save the answer.

Our advice is to start narrower than feels worthwhile. One document set, one type of question, sources visible on every answer, and a thumbs down button that stores the question alongside the passages that were retrieved. That log is the most valuable thing you will own after a month, because it tells you whether you have a retrieval problem, a content problem or a prompt problem, and those three need completely different fixes. Founders who treat RAG as a search project wearing an AI hat tend to ship something people keep using. Founders who treat it as a model problem tend to spend the budget tuning the wrong half.