Simple explanation
This lesson covers attacks that target the model itself — either corrupting it during training, or extracting information out of an already-trained model.
Technical explanation
- Data poisoning — corrupting a model's training data so it learns attacker-chosen, malicious behavior; happens during the data collection/training lifecycle stage.
- Model poisoning — directly tampering with a model's weights/parameters, rather than the training data that produced them; a different attack surface than data poisoning (attacking the output of training, not the input to it).
- Model skewing — gradually biasing a model's behavior over time through a stream of manipulated inputs, rather than one single attack — a slow-burn version of poisoning, harder to detect because no single input looks obviously malicious.
- Model inversion — reconstructing sensitive training data by carefully analyzing a model's outputs; a privacy attack against the training data itself, executed entirely through the model's normal output interface.
- Membership inference — determining whether a specific record was part of a model's training set at all, without necessarily reconstructing the record's full contents — a narrower privacy leak than full model inversion, but still a real one (e.g., confirming a specific patient's record was used to train a medical model).
- Model theft — extracting a functionally equivalent copy of a proprietary model, often by querying it extensively and training a new model on its outputs (a technique sometimes called "distillation" when used maliciously against someone else's model without authorization).
Synonyms / related terms
| Term | Means | |---|---| | Model extraction | Another name for model theft | | Distillation | The general technique of training a smaller model on a larger one's outputs, which can be misused for unauthorized model theft |
Concept Check
"A researcher determines that a specific patient's medical record was used to train a healthcare AI model, without ever seeing the record's actual contents." This is membership inference, not model inversion — the distinguishing feature is that only the fact of inclusion was determined, not the record's actual content; model inversion would involve reconstructing the record's actual sensitive details.
Interview-style Q&A
Q: Why is model skewing harder to detect than a single-event poisoning attack? A: "A single poisoning event usually involves an anomalous, identifiable batch of bad training data — something a careful audit might catch. Model skewing spreads the manipulation across many individually unremarkable inputs over time, each one looking legitimate in isolation. The malicious effect only becomes visible in the model's cumulative behavioral drift, which requires monitoring trends over time rather than inspecting any single input."
Memory trick
"Poison the Data, Skew it Slowly, Invert to Steal Secrets, Infer Membership, Theft the Whole Model" — five attacks, each tied to its own verb, covering the full range of ways a model or its training data can be attacked.