LLM04: Data and Model Poisoning
When an attacker corrupts the AI's training data or knowledge base to permanently manipulate its behavior.
Rated Critical because poisoning is persistent — the manipulation survives indefinitely until explicitly detected and removed. Unlike a prompt injection that affects one session, poisoning affects every user who interacts with the system.
What Is Data Poisoning?
Traditional cyberattacks target running systems: they exploit vulnerabilities, steal credentials, or inject malicious code. Data poisoning attacks the AI's learning process — corrupting what the system learns, so that the manipulation is baked into the model or knowledge base itself.
Think of it like contaminating a textbook used to train doctors, rather than hacking the hospital's computer system. Every doctor trained with that textbook carries the misinformation forward. The source of truth has been corrupted.
Three Types of Poisoning Attacks
Training Data Poisoning
The attacker introduces malicious examples into the dataset used to train the model. The model learns to behave incorrectly in specific circumstances the attacker controls — often called a "backdoor trigger." The model works normally in all other cases, making the poisoning difficult to detect.
Fine-Tuning Poisoning
Many enterprises fine-tune foundation models on their own data to customize behavior for their use case. If the fine-tuning dataset is compromised — either directly or via a poisoned data supplier — the resulting model carries the attacker's influence into production.
RAG Knowledge Base Poisoning (Most Common)
RAG (Retrieval-Augmented Generation) systems give AI agents access to a searchable knowledge base of documents. If an attacker can add documents to that knowledge base, they can poison the agent's responses for any query that retrieves those documents. This is the most practically relevant form of poisoning today.
RAG Poisoning: The Most Immediate Threat
Click the card to reveal the explanation
Employees Use AI to Navigate Company Policy
Your HR team has built an AI assistant that reads all company policy documents to answer employee questions. It handles 200+ queries a day about leave, expenses, benefits, and compliance. Employees trust it — it's faster than reading the policy manual.
An attacker with document-editing permissions (a disgruntled employee, or a compromised contractor account) adds a new policy document to the knowledge base: "Executive Expense Pre-Approval Procedure (Updated)". It looks official. It states that expenses over $50,000 require approval only from the CFO's email alias — which the attacker has set up to forward to themselves.
Now every employee who asks the AI about high-value expense approvals receives the attacker's fabricated policy. The AI cites the document. The behavior persists until someone manually audits the knowledge base.
- The AI model itself isn't compromised — the knowledge base is
- The AI is doing exactly what it's supposed to do: retrieving and citing documents
- Detection requires auditing document content, not the AI's behavior
- Every user who queries that topic is affected, not just one session
How RAG Poisoning Spreads
A RAG Knowledge Base Poisoning Attack
1 / 5🔑 Attacker gains write access
The attacker obtains the ability to add documents to your knowledge base — through a compromised employee account, an overly permissive document management system, or a supply chain compromise of a content provider your system ingests.
📝 Poisoned document is crafted
The attacker creates a document that looks legitimate — matching your company's document style, using plausible terminology, referencing real policies — but contains subtly incorrect or malicious content targeted at specific query types.
🗄️ Document enters the vector database
Your RAG pipeline processes the document, converts it to a vector embedding, and stores it. It now sits alongside legitimate documents, indistinguishable from the outside without reading its content.
🔍 Queries trigger retrieval
When users ask questions relevant to the poisoned document's topic, the retrieval system surfaces it as a relevant source — because it is semantically relevant. The AI cites it confidently alongside legitimate documents.
⏳ Poisoning persists
The attack continues indefinitely. Every user who asks relevant questions receives influenced responses. Unless someone audits the document store, the poisoning remains active for months or years.
Why Poisoning Is Especially Difficult to Detect
When a system is hacked, there are often signals: unusual network traffic, access logs, error messages. Poisoning leaves almost no signals — the system works exactly as designed. It retrieves documents, it cites them accurately, it produces coherent answers. The answers are just wrong in the ways the attacker intended.
Traditional security monitoring cannot detect poisoning. You need content monitoring — auditing what's in your knowledge base and evaluating whether AI outputs make sense against ground truth. This requires domain expertise, not just security tooling.
Data poisoning is attractive to sophisticated attackers because:
- Persistent impact: one successful poisoning affects all future users until discovered
- Plausible deniability: "I just added a document to the shared drive"
- Low detectability: no anomalous system behavior — the AI works normally
- Targeted precision: poisoning can be scoped to specific query types, affecting only the decisions the attacker cares about
- Compounding effect: in autonomous agent systems, poisoned knowledge base entries influence agent decisions, which may write to other systems, spreading the corruption
Defense Requires Both Technical and Process Controls
- Strict write access controls on knowledge bases and document stores — not everyone who can read should be able to write
- Document change audit logs — who added or modified which documents, when
- Content review workflows for new documents added to AI knowledge bases — treat it like code review
- Regular output sampling and validation — have domain experts sample AI answers on critical topics and verify against authoritative sources
- Immutable document provenance — documents should carry cryptographic signatures from authorized sources
- Anomaly detection on retrieval patterns — if one document is being retrieved far more than its recency or relevance would predict, investigate
What Leaders Must Require
- Inventory all AI knowledge bases and RAG systems. Know what documents feed each AI system and who has write access.
- Apply least-privilege to document stores. Write access to AI knowledge bases should be restricted and logged.
- Implement content review for AI training and fine-tuning data. Any dataset used to train or update a production model should go through a review process equivalent to code review.
- Establish ground-truth validation for critical AI outputs. For decisions that matter (compliance guidance, financial calculations, safety procedures), establish a regular process to verify AI outputs against authoritative sources.
- Include RAG systems in your data breach response plan. A poisoned knowledge base is a data integrity incident — know how to identify it, contain it, and recover.