LLM-Based Decision Making Framework for Autonomous Drone Navigation

Drones have seen a massive surge in popularity, driven by technological advancements, falling costs and versatility. Despite this, control schemes for drones are still a challenge to learn for most people, one that needs extensive time and training to master. Additionally, requiring an individual operator for every deployed drone limits the number of drones and the speed with which they can be deployed for a given task. Large Language Models (LLMs) are instrumental in passing this hurdle. By integrating their natural language understanding, it is possible to simplify their control scheme for use by a layman, and even to fully automate their operation by assigning an LLM as the operator itself. This paper presents an LLM driven framework for autonomous drone operation. The LLM is assigned the role of the drone operator, responsible for reasoning and decision making. A YOLOE-11L model is responsible for object detection and generating visual encodings to be interpreted by the LLM. The LLM generates actions to be taken in the environment by reasoning with the visual input and the context of the previously taken actions, which are passed along to a PX4 flight control stack. The test environment is a real life environment photogrammatically recreated in a high fidelity simulation in Unreal Engine 5. To evaluate the system, a representative mission "Land on a couch" is performed multiple times to test the robustness of the system. Experimental results show that the system can effectively interpret sensor data, perform spatial reasoning and perform safe trajectories without domain specific training.