RAG systems fail in production not because retrieval is broken, but because the prompt doesn't tell the model how to handle conflicting, incomplete, or irrelevant chunks. These three ready-to-use templates fix that by forcing explicit reasoning over retrieved context, and they work reliably across Claude, GPT-4, Gemini, and DeepSeek.
RAG Prompt Design is the difference between a chatbot that hallucinates confidently and one that admits when it doesn't know. I've tested these templates across GPT-4, Claude 3.5 Sonnet, Gemini 1.5 Pro, and DeepSeek-V3 with consistent results. The core issue is that most developers paste a generic "answer based on context" instruction and hope for the best. That works in demos and fails in production.
Why Generic Prompts Fail Here
Generic prompts like "Use the context to answer the question" fail because they don't specify how to handle the three most common RAG failure modes:
- Conflicting chunks — two retrieved passages say opposite things. The model picks one silently or blends them into nonsense.
- Irrelevant retrieval — the top-k chunks are tangentially related but don't actually answer the question. The model forces an answer anyway.
- Partial coverage — the answer requires information from 3 chunks but only 2 were retrieved. The model fills the gap with training data.
A well-designed RAG prompt explicitly instructs the model to detect these situations and respond accordingly. That's the entire job.
Template 1: The Grounded Answerer
This is your default template for standard Q&A over retrieved documents. It works best when you have decent retrieval quality (top-5 accuracy above 70%).
You are a precise research assistant. Answer the user's question using ONLY the provided context chunks.
CONTEXT:
{context_chunks}
QUESTION: {user_question}
RULES:
1. If the context contains the answer, cite the chunk IDs in brackets, like [C2][C4].
2. If the context is irrelevant to the question, respond: "The retrieved documents do not contain information relevant to this question."
3. If the context has conflicting information, present both perspectives and note the conflict explicitly.
4. Never use knowledge from your training data to fill gaps. If the context is insufficient, say so.
5. Keep your answer under 150 words unless the question requires deeper analysis.
ANSWER:
Placeholders:
{context_chunks}— your retrieved passages, prefixed with chunk IDs like[C1] Document: ...{user_question}— the raw user query, unmodified
Template 2: The Synthesis Engine
Use this when the answer requires combining information across multiple chunks. This is for multi-hop questions where no single chunk contains the full answer.
You are a legal analyst synthesizing evidence. The user's question requires combining facts from multiple sources.
CONTEXT (each chunk is labeled with source metadata):
{context_chunks}
QUESTION: {user_question}
TASK:
1. Identify which chunks are relevant to the question. List them as "RELEVANT: [C1], [C3]".
2. Identify which chunks are NOT relevant. List them as "IGNORED: [C2]".
3. Synthesize an answer that traces each claim back to its source chunk.
4. If the answer requires a fact that no chunk provides, state: "MISSING INFORMATION: [specific fact needed]".
5. Format your final answer as a structured brief with bullet points, each ending with a source citation.
FINAL ANSWER:
This template forces the model to show its reasoning chain before answering, which makes debugging retrieval failures dramatically easier. When an answer is wrong, you can see exactly which chunk it used.
Template 3: The Edge-Case Defender
For production systems where wrong answers are costly — medical, legal, or financial contexts. This template handles the worst case: retrieval returns garbage and the model must not hallucinate.
You are a conservative document analyst. Your primary job is to NOT make things up.
CONTEXT:
{context_chunks}
QUESTION: {user_question}
STRICT PROTOCOL:
1. First, determine relevance: Does ANY chunk directly address the question? If no, output: "UNABLE TO ANSWER: No retrieved document addresses this question." Stop there.
2. If chunks are partially relevant, extract only the directly relevant sentences. Quote them verbatim.
3. If chunks contradict each other, output: "CONTRADICTION DETECTED" and list the conflicting statements side by side.
4. NEVER infer, extrapolate, or combine facts from different chunks unless the combined conclusion is explicitly stated in a single chunk.
5. If your answer would be shorter than 20 words, just output the answer. Do not pad.
RESPONSE:
This template trades answer completeness for correctness. In my experience, it reduces hallucination rates from ~15% down to under 2% on adversarial retrieval sets.
How to Adapt These for Your Own Codebase
The templates are useless if you don't wire them into your pipeline correctly. Three concrete adjustments:
Inject metadata into chunks. Don't just paste raw text into {context_chunks}. Prepend source, date, and confidence scores: [C1 | Source: api_docs_v2 | Updated: 2025-01-15]. The model uses this to judge freshness and authority.
Set a dynamic token budget. If your retrieved context is 4,000 tokens, tell the model "The context is {context_length} tokens. Prioritize precision over completeness." Long contexts make models lazy about actually reading.
Log the prompt, not just the answer. Store the exact prompt template version and the chunk IDs you passed. When an answer is wrong, you need to know if it was a retrieval failure or a generation failure. This separation is critical for debugging.
Do These Prompts Work With Any LLM?
Yes, with minor caveats. I've tested all three templates on GPT-4, Claude 3.5 Sonnet, Gemini 1.5 Pro, and DeepSeek-V3. The core instructions — citation format, contradiction handling, and refusal behavior — work consistently across all four. The differences are stylistic: Claude tends to be more verbose, GPT-4 follows the "stop early" instruction best, and DeepSeek occasionally ignores the "never use training data" rule. If you're building on a smaller open-source model like Llama-3-8B, expect weaker instruction following — you'll need to simplify the templates to 3 rules max.
The One Adjustment That Improves These Prompts the Most
Add a single line at the end of any of these templates: "If the context is insufficient, state exactly what information is missing and what retrieval query would find it." This turns a silent failure into a signal you can use to improve your retrieval pipeline. Every time the model tells you what's missing, you get a free query reformulation hint. I've seen this one line cut RAG debugging time by half across multiple production systems.