Simple explanation
Generative AI — and specifically Large Language Models — is the technology behind nearly every AI security conversation happening right now. This lesson covers what it actually is under the hood.
Technical explanation
- Large Language Model (LLM) — a model trained on massive amounts of text data to generate human-like language, typically built on the transformer architecture.
- Transformer architecture — the neural network design (using a mechanism called self-attention) underlying most modern LLMs, allowing the model to weigh how relevant different parts of the input are to each other when generating output.
- Prompt engineering — the practice of carefully crafting input text to reliably elicit a desired output from an LLM; this is also directly relevant to security, since the same skill used constructively for prompt engineering is what an attacker uses destructively for prompt injection.
- Inference — the process of running a trained model on new input to produce output (using the model), as distinct from training it.
- Natural Language Processing (NLP) — the broader field of AI concerned with enabling computers to understand, interpret, and generate human language; LLMs are the current dominant approach to NLP, but NLP as a field predates them.
- Context window — the amount of text (measured in tokens) an LLM can consider at once when generating a response; relevant to security since anything within the context window can potentially influence output, including maliciously injected content.
Synonyms / related terms
| Term | Means | |---|---| | LLM | Large Language Model | | Self-attention | The core transformer mechanism weighing relevance between input parts | | Context window | The amount of input text an LLM can consider at once |
Concept Check
"A security team wants to understand why a model can be tricked into ignoring its safety instructions by cleverly-worded input placed earlier in the same conversation." This connects directly to the context window and self-attention — everything within that window, including earlier conversation turns, can influence how the model weighs and responds to later input, which is exactly the mechanism prompt injection exploits.
Interview-style Q&A
Q: Why does understanding prompt engineering matter for someone focused on AI SECURITY rather than AI development? A: "Prompt injection is fundamentally malicious prompt engineering — the same skill of carefully crafting input to control model output, just applied with harmful intent instead of legitimate use. Understanding how prompt engineering actually works to influence a model's behavior is what lets a security practitioner recognize and defend against its adversarial version."
Memory trick
"Transform the Attention, Generate the Language" — a quick tag linking the transformer architecture's core mechanism (attention) to what it ultimately produces (generated language).