RAG reduces hallucinations, but it does not eliminate them. The biggest trust gains often come from a simple capability: knowing when not to answer. An answerability gate is the decision logic that chooses between answering, asking a clarifying question, or abstaining.
Use retrieval confidence, not model confidence
Most assistants can sound confident even when retrieval is poor. Anchor gating on evidence quality:
- Coverage. Do retrieved sources mention the key entities and constraints in the question?
- Agreement. Are sources consistent or do they contradict each other?
- Specificity. Do you have exact values, dates or policy text, or only general guidance?
- Freshness. Is the source recent enough for the question (see freshness evaluation)?
If retrieval quality is low, the best prompt engineering in the world will not save the answer. Start by improving retrieval (see retrieval quality and ranking and relevance).
Ask clarifying questions when the query is underspecified
Many failures are not knowledge failures; they are ambiguity failures. Ask when:
- The user did not provide a timeframe ("current" vs "last quarter").
- Multiple policies apply (region, product tier, customer type).
- The user intent is unclear (explain vs decide vs draft).
A good clarifying question should be short, present 2-3 options, and explain why it matters. This improves answerability without forcing a refusal.
Abstain with a safe fallback when evidence is missing
When you cannot support a reliable answer, abstain in a user-helpful way:
- Say what you can confirm. Summarise what sources do support.
- Show the gap. Name the missing document, system or timeframe.
- Offer next steps. Ask for context, propose who to contact, or suggest a search.
For high-risk domains, abstention should be the default unless evidence is strong and policy allows it (see guardrails).
Make gates observable and tune them like a product
Gates are thresholds, and thresholds need tuning. Instrument:
- Answer rate. How often you answer vs ask vs abstain.
- Escalation rate. How often abstention leads to a useful handoff.
- Quality by gate path. Compare answers that passed high confidence vs low confidence.
Over time you want fewer low-confidence answers, fewer unnecessary refusals, and better outcomes when you do abstain.
Answerability gates are not about being cautious. They are about being honest with evidence and consistent with policy. That is how you earn long-term trust.