The Adversary's Playbook

This chapter presents a detailed exploration of how adversaries identify, develop, and operationalize attacks against machine learning systems. Through a structured analysis of attacker profiles and access levels—white-box, gray-box, and black-box—it uncovers how the depth of model visibility shapes the techniques used to evade, poison, or manipulate artificial intelligence (AI) systems. The chapter emphasizes that modern adversaries are not limited by traditional technical boundaries; they are agile, persistent, and increasingly coordinated in exploiting weaknesses across the AI lifecycle. Readers are guided through real-world adversarial tactics such as gradient-based evasion, surrogate modeling, prompt injection, membership inference, and multi-stage poisoning campaigns. Special focus is given to the evolution of attacks from isolated experiments into persistent, lifecycle-aware operations that embed triggers, influence retraining cycles, and bypass safety mechanisms. The content demonstrates how adversaries chain vulnerabilities across phases—data collection, model tuning, inference, and retraining—to create resilient exploits that evade common defenses. The chapter reinforces the strategic importance of viewing AI security through an adversary-centric lens. It equips professionals with the mental models needed to anticipate attacker behavior, prioritize defensive monitoring, and align security controls with real-world threat patterns. By understanding the adversary's playbook, AI practitioners and cybersecurity teams can more effectively protect systems against sophisticated, long-term manipulation and exploitation.