RAG & Knowledge Bases

The confident stranger

Ask a model about your company's refund policy and it will answer. Fluently. With structure and caveats and a helpful closing line. And it will be making the whole thing up, because it has never seen your refund policy.

The model knows an enormous amount about the world in general and nothing whatsoever about you — your documents, your tickets, your wiki, last Tuesday's decision. Worse, it can't tell the difference between knowing and not knowing, so the invented answer arrives with exactly the same confidence as a real one.

The obvious fix is to paste your documents into the prompt. That works until you have more than a handful: it gets slow, expensive, and eventually exceeds the context window. And it's wasteful — a question about refunds doesn't need your engineering handbook.

RAG is just this idea: don't send everything, send the few pieces that are actually relevant to the question being asked.

Retrieval-Augmented Generation. Retrieve the relevant bits, augment the prompt with them, let the model generate from there. The hard part — the part worth understanding — is how the machine decides what's relevant.

What embeddings actually do

Searching for keywords fails immediately here. Someone asking "how do I get my money back?" won't match a policy that says "returns are eligible for a full refund" — not one word in common, same meaning exactly.

An embedding solves this by turning text into a list of numbers — a few hundred to a few thousand of them — positioned so that things that mean similar things end up close together. Comparing two pieces of text becomes comparing two points in space.

Pick a question and watch which documents rise. Then hit the keyword toggle to see what a plain text search would have found instead:

Chunking is where most RAG systems are won or lost

You can't embed a whole 40-page handbook as one vector. Squash a document that size into a few hundred numbers and you get the average of everything it discusses, which is close to nothing in particular. So documents get split into chunks, and each chunk gets its own embedding.

How big? That single number quietly determines how well the whole system works. Drag it:

Two failure modes sit at the ends of that slider. Too small and an idea gets cut in half, so neither piece makes sense alone. Too large and one chunk covers three unrelated topics, so its embedding is a blur and it matches everything vaguely and nothing well.

The standard defence is overlap — let each chunk repeat the last sentence or two of the one before it, so an idea that straddles a boundary survives intact in at least one chunk. It costs a little storage and prevents a whole category of "why didn't it find that?" bugs.

The pipeline, end to end

Two phases. Ingestion happens once, ahead of time: split documents, embed each chunk, store the vectors. Query happens per question, and it's the loop you'll actually debug:

Two details in there cause most of the bugs. The question must be embedded with the same model used at ingestion — mix two models and you're comparing coordinates from different maps. And the final prompt should tell the model to answer only from the supplied chunks and to say so when they don't cover it; without that instruction it will happily fall back on its own training data, which is the exact problem you built this to avoid.

When the answer is wrong

RAG fails in a specific, diagnosable way: almost always, retrieval brought back the wrong material and the model did its best with it. Before you touch the prompt or swap models, look at what got retrieved. Three cases:

What you've got now

A model that can answer from your material instead of its memory, with citations you can check. That covers an enormous share of real AI products — support assistants, documentation search, internal Q&A, research tools.

But notice what it still can't do: act. It reads and answers. It can't file the ticket, update the record, or run the job. That's next.