RAG systems inherit the risks of their content supply chain. If an attacker (or a well-meaning but mistaken contributor) can introduce bad content into the knowledge base, the assistant will confidently repeat it. This is data poisoning: the system becomes wrong because its sources are wrong.
Poisoning is not always a dramatic hack. In many enterprises, the most common “poisoning” is low-quality or outdated material that is indexed without ownership, freshness rules, or review. The mitigation is the same: treat ingestion as a governed pipeline.
What poisoning looks like in RAG
- Malicious instructions embedded in documents. Variants of prompt injection hidden in policies, wikis, or PDFs (see prompt injection defence).
- Subtle misinformation. Small changes to procedures, thresholds, or terms that are hard to spot.
- Source swapping. A link or canonical document is replaced, so retrieval pulls the wrong “truth”.
- Content drift. Old guidance remains indexed and competes with updated guidance.
Control the sources, not just the model
Defences start with the simplest control: a source-of-truth model. Keep an allow-list of approved sources and require owners for each source (see knowledge base governance). If a source has no owner, it should not be indexed.
For high-risk domains, treat sources like software releases:
- Versioned ingestion. Snapshot content and log which snapshot powered each response.
- Change approvals. Require review for policy-critical documents.
- Rollback. If a bad batch is ingested, revert quickly (see incident response).
Build ingestion-time scanning and classification
Do not index everything blindly. Add ingestion checks (see RAG ingestion pipelines):
- Content classification. Detect restricted content and route it to controlled indexes or exclude it.
- Prompt-injection heuristics. Flag “ignore previous instructions”, credential requests, and other common patterns.
- Duplicate and staleness detection. Reduce competing versions of the same guidance.
- Provenance metadata. Record owner, timestamps, and source system IDs for every chunk.
Detect poisoning with evaluation and monitoring
Poisoning often shows up as a shift in answer quality on specific intents. Use a small set of golden queries and run them continuously against the latest corpus (see retrieval quality and evaluation loops). Watch for:
- Sudden citation changes for stable questions.
- Lower citation relevance or coverage.
- New answers that contradict established sources.
Pair this with drift monitoring so you can spot changes early (see drift monitoring).
Containment and recovery
When poisoning is suspected, treat it like an incident: freeze ingestion, roll back the corpus snapshot, and shift to evidence-first or escalation modes until the root cause is addressed. The goal is to preserve trust while you restore integrity.
RAG makes knowledge accessible at scale. That only works when the knowledge pipeline is treated as a governed, defendable system—not a convenient dump of documents.