Building a RAG Pipeline from Scratch

Why build RAG yourself

Retrieval-augmented generation sounds like a big piece of infrastructure, but at its core it's a small pipeline solving one problem: your LLM doesn't know about your data, and the context window is too small (or too expensive) to paste everything in every time. RAG fixes that by retrieving only the handful of chunks that are actually relevant to the question, then handing those to the model along with the question itself.

Every managed RAG product — vector databases with built-in retrieval, framework abstractions, hosted pipelines — is doing the same five steps under the hood. Building it yourself once, even with a tiny in-memory version, makes every one of those products click. That's what this post walks through: chunking, embedding, storing, retrieving, and streaming an answer back into a chat UI. No framework, just fetch calls and arrays.

Step 1: Chunking your documents

You can't embed an entire document as one vector and expect useful retrieval — a 20-page PDF squashed into a single embedding loses almost all of its specificity. Instead you split it into chunks small enough to be topically focused, but large enough to still make sense out of context.

A good starting point is 300–500 tokens per chunk with a small overlap between consecutive chunks, so an idea that straddles a chunk boundary doesn't get cut in half:

function chunkText(text, chunkSize = 800, overlap = 100) {
  const chunks = [];
  let start = 0;

  while (start < text.length) {
    const end = Math.min(start + chunkSize, text.length);
    chunks.push(text.slice(start, end));
    start += chunkSize - overlap;
  }

  return chunks;
}

This is a character-based splitter, which is fine to start with. In production you'll want to split on sentence or paragraph boundaries instead, so you don't cut a sentence in half mid-word — but the shape of the problem is identical.

Step 2: Generating embeddings

An embedding is a vector — an array of a few hundred to a few thousand floats — that represents the meaning of a piece of text. Texts with similar meaning end up with vectors that point in similar directions, which is the property retrieval depends on. You get embeddings by calling an embedding model's API, one request per chunk (or batched):

async function embed(text) {
  const response = await fetch('https://api.openai.com/v1/embeddings', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${process.env.OPENAI_API_KEY}`
    },
    body: JSON.stringify({
      model: 'text-embedding-3-small',
      input: text
    })
  });

  const data = await response.json();
  return data.data[0].embedding; // e.g. [0.0123, -0.0456, ...]
}

Run this once per chunk when you ingest a document, and store the resulting vector next to the chunk's text. This is the expensive, one-time cost of RAG — embedding happens at ingestion time, not at query time, which is why RAG stays fast even over large document sets.

Step 3: Storing the vectors

At small scale — a few thousand chunks — you don't need a dedicated vector database at all. A plain array in memory (or a JSON file on disk) works fine, and it makes the mechanics of retrieval much easier to see:

const vectorStore = []; // { id, text, embedding, sourceDoc }

async function ingest(docId, rawText) {
  const chunks = chunkText(rawText);

  for (const [i, chunk] of chunks.entries()) {
    const embedding = await embed(chunk);
    vectorStore.push({
      id: `${docId}-${i}`,
      text: chunk,
      embedding,
      sourceDoc: docId
    });
  }
}

Step 4: Retrieval — finding the relevant chunks

At query time, you embed the user's question with the exact same embedding model, then compare that query vector against every stored chunk vector using cosine similarity — a measure of how closely two vectors point in the same direction, from -1 (opposite) to 1 (identical):

function cosineSimilarity(a, b) {
  let dot = 0, normA = 0, normB = 0;

  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    normA += a[i] * a[i];
    normB += b[i] * b[i];
  }

  return dot / (Math.sqrt(normA) * Math.sqrt(normB));
}

async function retrieve(question, topK = 4) {
  const queryEmbedding = await embed(question);

  return vectorStore
    .map(item => ({ ...item, score: cosineSimilarity(queryEmbedding, item.embedding) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, topK);
}

That's the entire "retrieval" in retrieval-augmented generation: one embedding call for the question, a similarity score against every stored vector, and the top few by score. A real vector database does this same comparison faster using an approximate nearest-neighbor index instead of a brute-force scan — but the result you get back is conceptually the same.

Retrieval quality is almost always the actual bottleneck in a RAG system, not the language model. If answers feel wrong or generic, check what got retrieved before you touch the prompt or swap models.

Step 5: Assembling the prompt

With the top chunks in hand, you build a prompt that gives the model the retrieved context and instructs it to answer using only that context — which is what keeps RAG grounded instead of letting the model fall back on its own (possibly outdated, possibly wrong) training data:

function buildPrompt(question, retrievedChunks) {
  const context = retrievedChunks
    .map((c, i) => `[${i + 1}] ${c.text}`)
    .join('\n\n');

  return `Answer the question using only the context below. If the context
doesn't contain the answer, say you don't know — don't guess.

Context:
${context}

Question: ${question}

Answer, citing sources like [1] where relevant:`;
}

Step 6: Streaming the answer into a chat UI

The last step is sending that assembled prompt to the model and streaming the response back token-by-token, so the user sees words appear as they're generated instead of waiting for the full answer. On the client, that means reading the response body as a stream rather than waiting for response.json():

async function askQuestion(question, onToken) {
  const chunks = await retrieve(question);
  const prompt = buildPrompt(question, chunks);

  const response = await fetch('/api/chat', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ prompt, stream: true })
  });

  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    onToken(decoder.decode(value, { stream: true }));
  }
}

// Usage in the UI:
askQuestion('How do refunds work?', (token) => {
  messageEl.textContent += token;
});

The /api/chat endpoint here is your own server-side route — it holds the API key, forwards the prompt to the model with streaming enabled, and pipes the raw stream straight through to the client. Keeping the model call server-side, rather than calling the provider directly from the browser, is what keeps your API key out of the client bundle.

Common pitfalls

A few things trip up most first RAG builds, roughly in the order you'll hit them:

Chunks too big or too small. Too big and irrelevant text dilutes the similarity score; too small and you lose the surrounding context the model needs to answer well. Start around 500 tokens and adjust based on what actually gets retrieved.

No overlap between chunks. Without overlap, an answer that spans a chunk boundary can end up split across two chunks that individually don't look relevant enough to retrieve.

Retrieving too few or too many chunks. Four to six is a reasonable default. Too few and you miss the answer; too many and irrelevant context crowds out the useful part of the prompt.

Where to go from here

Everything above fits comfortably in a couple hundred lines with no framework: chunk, embed, store, retrieve, prompt, stream. That's deliberate — understanding the pipeline at this level means every abstraction built on top of it (LangChain, LlamaIndex, a managed vector database's built-in retrieval) reads as a convenience wrapper around steps you already understand, not a black box.

The next step is swapping the in-memory array for a real vector store once your document count grows, and tuning chunk size, overlap, and top-K against your own content rather than the defaults used here. Those three knobs account for most of the difference between a RAG system that feels magical and one that feels like it's guessing.