The Generative AI Attack Surface

This chapter explores the unique and expanding attack surface of generative artificial intelligence (AI) systems, particularly large language models, highlighting how their architectural design introduces vulnerabilities distinct from traditional predictive models. By examining key components such as transformer attention mechanisms, token embeddings, autoregressive output patterns, and contextual conditioning, the chapter reveals how adversaries can exert influence without needing access to source code or system internals. The inherent flexibility and openness of generative outputs make these systems highly susceptible to manipulation through subtle prompt engineering, data poisoning, and internal activation triggers. The chapter also delves into the risks posed by retrieval-augmented generation (RAG) systems, where external documents are dynamically incorporated into model context during inference. These integrations open the door to indirect prompt injection, corpus poisoning, and semantic redirection—often without triggering visible model misbehavior during standard evaluations. Fine-tuning, especially when performed with unvetted or third-party data, is shown to be a powerful vector for embedding persistent backdoors and trigger-based behaviors that evade traditional QA processes but remain reliably exploitable in production environments. For professionals securing AI applications, this chapter reframes the threat model around generative systems as a dynamic behavioral risk, not just a static software flaw. It emphasizes the importance of understanding internal model structures, context-aware manipulation techniques, and adversarial testing that mirrors how attackers condition models across time, input structure, and retrieved data. With clear analysis and applied framing, this chapter equips readers to identify, evaluate, and begin defending against the behavioral exploits that define modern generative AI threats.