α 3 -Bench: A Unified Benchmark of Safety, Robustness, and Efficiency for LLM-Based UAV Agents Over 6G Networks

Large Language Models (LLMs) are increasingly being used as high-level controllers for autonomous Unmanned Aerial Vehicle (UAV) missions. However, current evaluations rarely assess whether such agents remain safe, protocol-compliant, and effective under realistic constraints of next-generation networking. This paper introduces $\alpha ^{3}$ -Bench, a benchmark for assessing LLM-driven UAV autonomy as a multi-turn conversational reasoning and control problem operating under dynamic 6G conditions. Each mission is modelled as a language-mediated control loop between an LLM-based UAV agent and a human operator, where decisions must satisfy strict schema validity, speaker alternation, mission policies, and safety bounds while adapting to fluctuating network slices (URLLC/eMBB/mMTC), latency, jitter, packet loss, throughput, and edge-load variations. To reflect modern agentic workflows, $\alpha ^{3}$ -Bench integrates a dual-action layer that supports both Model Context Protocol (MCP) tool calls and Agent-to-Agent (A2A) coordination, enabling the evaluation of tool-use consistency and multi-agent interactions within UAV missions. We construct a large-scale corpus of 113k AI conversational episodes grounded in UAVBench scenarios and evaluate 17 state-of-the-art LLMs using a fixed subset of 50 episodes per scenario under deterministic decoding. We propose a composite $\alpha ^{3}$ metric that unifies six pillars—Task Outcome, Safety Policy, Tool Consistency, Interaction Quality, Network Robustness, and Communication Cost—with reliability- and efficiency-normalized scores per second and per thousand tokens. Experimental results show that several frontier models achieve near-perfect mission success and safety compliance (Task Outcome and Safety Policy $\geq 0.95$ ), yet their robustness and efficiency diverge substantially: under degraded 6G conditions, Network Robustness scores drop by up to 30–40%, while efficiency-normalized performance varies by more than a $2\times $ factor across models in both $\alpha ^{3}$ per-second and $\alpha ^{3}$ per-1k-tokens metrics, reflecting significant variance in inference latency and token consumption. $\alpha ^{3}$ -Bench provides a reproducible, extensible foundation for benchmarking conversational UAV autonomy and guiding the development of safe, network-aware, and resource-efficient LLM agents in 6G-enabled aerial systems. To support open science and reproducibility, we release the $\alpha ^{3}$ -Bench dataset on GitHub: https://github.com/maferrag/AlphaBench