Simple explanation
This lesson covers three risks specifically about how AI systems are INTEGRATED into larger systems and workflows — not attacks on the model itself, but on how much trust and autonomy it's given.
Technical explanation
- Excessive agency — an AI agent granted more permissions or autonomy than its task actually requires, so a manipulated prompt can cause outsized real-world damage. Example: an AI coding assistant with auto-merge permissions on pull requests, manipulated via a crafted PR description into merging malicious code.
- Insecure output handling — trusting an LLM's output and passing it downstream (into a database query, a shell command, rendered HTML) without validation — treating AI-generated output as inherently safe, the AI-era equivalent of trusting unvalidated user input.
- Overreliance — humans trusting AI-generated output without verification, even when it's hallucinated, biased, or simply wrong; a human-factors risk rather than a purely technical one, but a real security concern when it means skipped review of AI-suggested code, AI-drafted incident response actions, or similar.
- AI supply chain attack — a compromised third-party pre-trained model, dataset, or library carries its compromise forward into any system built on top of it; the AI-specific version of the general software supply chain risk, but with the added wrinkle that a compromised MODEL's malicious behavior can be far harder to detect through code review than compromised traditional software, since the "logic" lives in learned weights rather than inspectable source code.
Synonyms / related terms
| Term | Means | |---|---| | Excessive agency | An AI agent with more permission/autonomy than its task requires | | Insecure output handling | Trusting AI output without validation before downstream use | | AI supply chain attack | A compromised upstream model, dataset, or library propagating to downstream systems |
Concept Check
"An AI-generated shell command is executed automatically by an automation pipeline without any human or automated review of what the command actually does." This is specifically insecure output handling, not excessive agency — the pipeline's fundamental problem is that it never validates the AI's OUTPUT before acting on it; excessive agency would describe a scenario about the AI's own granted PERMISSIONS being too broad, a related but distinct issue.
Interview-style Q&A
Q: Why is a compromised open-source pre-trained model harder to detect than compromised traditional open-source code? A: "Traditional malicious code can often be spotted through source review — the logic is human-readable, even if obfuscated. A compromised model's malicious behavior lives in its learned weights, which aren't human-interpretable in any straightforward way. You can't 'read' a model's weights the way you'd read a function to spot a backdoor; detecting a poisoned model generally requires behavioral testing rather than static inspection, which is a fundamentally harder problem."
Memory trick
"Agency grants too much, Output trusts too much, Supply chain inherits too much" — three risks, each defined by trusting something too broadly: permissions, unvalidated output, or an unvetted upstream source.