A System for Procedural Generation of Virtual Agent Activity Using LLM-Based Scene Interpretation

This thesis presents a system for procedurally generating interactive stories and actions for virtual characters within Unity using large language models (LLMs). Users could drag and drop multiple agents and objects into a scene, with each entity automatically assigned semantic metadata describing its identity, role, and potential interactions. The scene structure is then serialized into a natural language prompt and sent to an LLM-based system, which returns a structured string describing a sequence of actions and interactions among agents and objects. The returned string encodes who performed which actions, when, and how, capturing layered behaviors such as walking, grabbing, sitting, toggling, or performing simple interactions using inverse kinematics (IK) control. A custom parser interprets this string and triggers coordinated agent behaviors using Unity’s NavMeshAgent navigation, animation layers, and interaction modules. The system supports simultaneous multi-agent scenes, dynamic object manipulation, and diverse interaction types without the need for manual scripting or pre-authored behavior trees. Designed for ease of use and rapid iteration, the system enables scalable, AI-assisted storytelling suitable for prototyping simulations and cinematic sequences. The performance of the developed system was evaluated using four popular lightweight LLMs, including OpenAI’s ChatGPT, Google’s Gemini, Anthropic’s Claude, and xAI’s Grok. Each model’s process and response time were measured under multiple complexity scenarios. The collected data were analyzed to compute and compare average response times and consistency across the examined scenarios, highlighting the relative efficiency and suitability of each model for procedural narrative generation. The results demonstrate that large language models can reliably translate high-level scene descriptions into executable multi-agent behaviors suitable for real-time environments. Overall, the system establishes a practical foundation for integrating natural-language narrative planning with procedural character interaction in Unity.