A Cascaded Hybrid Architecture for Automated Penetration Testing: Deep Reinforcement Learning, Expert-Guided Imitation Learning, and LLM-Based Reasoning

Abstract Automating penetration testing requires agents that can plan and act under uncertainty, operate over large state-dependent action spaces, and learn effectively from sparse feedback, yet existing single-policy approaches often fail to scale as scenario complexity increases. To address this problem, this thesis proposes a cascaded hybrid architecture that combines deep reinforcement learning (DRL), expert-guided imitation learning (IL), and constrained largelanguage- model (LLM) reasoning within a central Cascaded Policy Orchestrator (CPO). The CPO adaptively routes control among a PPO-based DRL policy, an IL policy equipped with domain guardrails and abstention for out-of-distribution states, and a budgeted LLM planner restricted to a validated Top-K action set, using lightweight confidence- and progress-based gating to escalate or return control as needed. The approach is evaluated in the NASim environment across canonical scenario scales (tiny, small, medium, large-gen, huge-gen, pocp-1-gen) against DRL-only, IL-only, and LLM-only baselines, with ablation studies examining the effects of routing logic, guardrails, abstention, and hand-back criteria. Results show that the cascaded agent matches baseline performance on compact scenarios and achieves higher success with fewer steps on large, huge, and pocp-1 environments, while maintaining moderate negative-reward rates and confining LLM usage to short, predictable bursts; paired nonparametric tests indicate statistically significant gains with large effect sizes on the hardest scenarios. These findings demonstrate that adaptive policy orchestration can improve scalability, efficiency, and cost control in autonomous penetration testing, providing a practical pathway toward reliable automation and motivating future validation in richer cyber-range settings.