Most RAG systems that underperform aren't failing because of the generation model — they're failing because of retrieval quality, and no amount of prompt engineering on the generation side fixes a system that's retrieving the wrong chunks of context in the first place.
Retrieval-augmented generation (RAG) combines a retrieval step — finding relevant information from a knowledge base using semantic search over embeddings — with a generation step, where a model produces an answer grounded in the retrieved content, rather than relying solely on what the model learned during training.
Why RAG Matters (and When Fine-Tuning or Long Context Fits Better)
RAG matters for knowledge that changes frequently or is too large to fit in a prompt or a model's training data — documentation, internal knowledge bases, or any corpus that needs to stay current without retraining a model every time it changes, where retrieval at query time naturally reflects the latest content.
Fine-tuning or long-context approaches fit better for knowledge that's relatively stable and needs to shape the model's behavior or style rather than just inform specific facts, or for corpora small enough to fit entirely within a large context window without needing selective retrieval at all — RAG's retrieval step adds complexity that isn't warranted if the whole knowledge base can simply be included directly.
Getting Started with a RAG System
A basic RAG pipeline: chunk, embed, retrieve, generate.
async function ingestDocument(text: string, docId: string) {
const chunks = chunkText(text, { size: 500, overlap: 50 });
for (const chunk of chunks) {
const embedding = await embedText(chunk);
await vectorDb.insert({ docId, chunk, embedding });
}
}
async function answerQuestion(question: string) {
const questionEmbedding = await embedText(question);
const relevantChunks = await vectorDb.search(questionEmbedding, { topK: 5 });
const context = relevantChunks.map((c) => c.chunk).join("\n\n");
return await model.generate({
messages: [{
role: "user",
content: `Context:\n${context}\n\nQuestion: ${question}\n\nAnswer based only on the context above.`,
}],
});
}
Core RAG Concepts Every Developer Should Know
Chunking strategy directly determines what's available to retrieve — chunks too large dilute relevance (retrieving a chunk with the right information buried among a lot of irrelevant text), while chunks too small lose necessary surrounding context. Chunk size and overlap are worth tuning against your actual content and query patterns rather than using a fixed default blindly.
Retrieval quality is the primary lever for overall RAG system quality, more so than generation model choice — a generation model working from irrelevant or incomplete retrieved context produces a poor answer regardless of how capable it is, since it can only work with what retrieval actually surfaced.
Embedding model choice affects how well semantic similarity actually captures relevance for your specific content and query style — a general-purpose embedding model may underperform a domain-specific one for specialized content (legal, medical, code), and this is worth evaluating explicitly rather than assuming any embedding model works equally well for any domain.
Grounding instructions in the generation prompt reduce (but don't eliminate) the risk of the model ignoring retrieved context and answering from its own training knowledge instead — explicitly instructing the model to answer only from provided context, and evaluating whether it actually does, is a meaningful quality lever distinct from retrieval quality itself.
Common Mistakes Building RAG Systems and How to Fix Them
Mistake 1: using a fixed chunk size without tuning against actual content and query patterns, producing chunks that are consistently too large or too small for good retrieval relevance. Fix: experiment with chunk size and overlap against representative queries, measuring actual retrieval quality rather than assuming a default fits your content.
Mistake 2: not evaluating retrieval quality separately from end-to-end answer quality, making it hard to tell whether a poor answer comes from bad retrieval or bad generation. Fix: evaluate retrieval quality independently (are the right chunks actually being retrieved) before attributing quality issues to the generation step.
Mistake 3: insufficient grounding instructions, letting the model blend retrieved context with its own training knowledge in ways that can introduce inaccuracies not actually supported by the retrieved content. Fix: explicitly instruct the model to answer only from provided context, and evaluate whether responses actually stay grounded rather than assuming the instruction alone guarantees it.
When Should You Use RAG Instead of a Long Context Window?
Use RAG when your knowledge base is too large to fit in a context window economically, changes frequently enough that re-embedding is cheaper than reprocessing a huge prompt every request, or when you need to scale to many documents where selective retrieval is more efficient than including everything. Use a long context window when your knowledge base is small enough to fit entirely and reasonably cheaply, and the operational simplicity of skipping a retrieval pipeline outweighs RAG's benefits for your specific scale.
RAG Systems in Production
Tune chunking strategy against actual content and query patterns rather than a fixed default, and evaluate retrieval quality independently from end-to-end answer quality to isolate where problems actually originate. Choose an embedding model suited to your specific content domain, and use explicit grounding instructions while actually verifying the model follows them.
If your RAG system's answers seem off, check retrieval quality first before touching the generation prompt — the majority of RAG quality problems trace back to retrieval surfacing the wrong or incomplete context, not the generation model failing to use good context well.