LLM09: Misinformation
When AI confidently generates false information — and your organization acts on it.
The Confident Wrong Answer
Every other vulnerability in this list involves an attacker deliberately doing something. Misinformation is different: it happens when the AI is working exactly as designed, with no attacker involved, and still produces output that causes real harm.
Large language models are trained to produce fluent, coherent, contextually appropriate text. They are not trained to be factually accurate. The distinction matters enormously. An LLM can produce confidently-stated nonsense with the same fluency and apparent authority as accurate information. It doesn't know the difference — and it doesn't signal the difference.
The industry term for this is "hallucination" — but that word undersells the business risk. Hallucination sounds like a minor glitch. Confidently fabricated legal citations, financial projections, or medical guidance that get acted upon are operational and legal liabilities.
Click the card to reveal the explanation
A Lawyer Uses AI to Research Case Precedents
A lawyer preparing a brief for a US federal court uses an AI assistant to research relevant legal precedents. The AI returns a detailed list of 6 citations: case names, courts, dates, judges, and key quotes. The output looks completely authoritative.
This is not a hypothetical. In 2023, in the real case of Mata v. Avianca in US federal court, a lawyer submitted an AI-generated brief containing fabricated case citations. The judge requested copies of the cited cases. They could not be produced because they did not exist. The court issued sanctions.
The AI produced realistic-sounding citations because it was trained on legal text and learned to produce text that sounds like legal citations. It had no mechanism to verify whether those cases actually existed.
- The AI's tone conveyed no uncertainty — there was no indication these citations were fabricated
- The lawyer trusted the output without independent verification
- The harm was professional (sanctions) and reputational, but could have been far worse in different contexts
- This exact failure pattern occurs in medical, financial, and regulatory contexts daily
Why LLMs Hallucinate
The Mechanism Behind AI Misinformation
1 / 4🔤 LLMs predict the next word, not facts
A language model is fundamentally trained to predict what text comes next given what came before. It learned from vast amounts of human text. When asked a question, it produces the text that most plausibly follows — which is usually accurate, but not always.
🎭 Confidence is a style, not a signal
The AI produces confident-sounding text because confident text appears more frequently in training data for authoritative topics. It learned to sound like an expert. But sounding like an expert and being correct are different things — the model doesn't know the difference.
🕳️ Gaps are filled with plausible invention
When the model doesn't have relevant training data for a specific fact, it doesn't say 'I don't know.' It fills the gap with text that fits the context — a plausible-sounding name, date, or figure. This is the hallucination.
🤝 The user's trust amplifies the risk
Research consistently shows humans over-trust AI output — called automation bias. The AI's confident, well-formatted, grammatically perfect output inspires more trust than it deserves. Verification rates are far lower than the error rates justify.
Where Misinformation Causes the Most Damage
Legal: Fabricated case citations, incorrect regulatory requirements, inaccurate contract interpretations. Consequences: sanctions, malpractice, compliance failure.
Financial: Incorrect financial projections, fabricated market data, wrong regulatory figures. Consequences: bad investment decisions, misstatements in filings, audit findings.
Medical: Wrong drug interactions, incorrect dosage information, fabricated research citations. Consequences: patient harm, liability, regulatory action.
Regulatory/Compliance: Incorrect interpretation of requirements. Consequences: non-compliance, fines, enforcement action based on wrong understanding.
Governance Controls for Misinformation Risk
- Define mandatory verification domains. For your organization, identify which topics require independent verification before AI output is acted upon (legal citations, financial figures, regulatory requirements, medical information). Write this into your AI usage policy.
- Scope AI to retrieval, not synthesis, for critical facts. An AI that retrieves and quotes from verified documents (with citations) is less prone to hallucination than one synthesizing from memory. Use RAG with trusted sources for high-stakes domains.
- Train employees on automation bias. The default human behavior is to trust confident AI output. Your employees need explicit training that AI confidence is not a reliability signal.
- Build verification into workflows, not reviews. "Have a human check the AI output" is advice that gets ignored under time pressure. Build verification steps into the workflow — required fields, source citation requirements, sign-off checkboxes before action.
- Monitor AI output quality. For high-volume, high-stakes AI applications, implement sampling and evaluation of output quality. Spot-check AI answers against verified sources on a regular cadence.