UDAQ: Unified Dynamic and Adaptive Q-iteration for Unknown Environment Path Planning and Exploration

Abstract Autonomous navigation in unknown environments requires both efficient point-to-point path planning and reliable full environment exploration. This paper presents UDAQ, a Unified Dynamic and Adaptive Q-iteration framework that achieves local path planning, complete environment exploration, and oriented exploration behavior using a single reward-driven formulation. Unlike conventional approaches that rely on separate planners or task-specific heuristics, UDAQ adapts its behavior through reward configuration within a unified value iteration framework. A key feature of UDAQ is its ability to operate under dynamically evolving state spaces, as encountered in SLAM-based mapping, where the environment expands and changes over time. To ensure stable convergence, an adaptive reward scaling mechanism is employed, enabling effective value propagation as the state space grows and preventing policy degradation. UDAQ functions as a fully autonomous closed-loop system, continuously integrating SLAM updates, adapting its state representation and reward parameters, performing Q-iteration to convergence, and generating navigation commands without human intervention. The method is evaluated in three simulated environments of increasing structural complexity and compared against the ROS 2 NavFn planner for point-to-point navigation, as well as frontier-based exploration, wall-following, and random walk for exploration tasks. In point-to-point navigation, UDAQ reduced path length by approximately 8–12% and travel duration by 20–30% compared to NavFn, approaching global optimal solutions despite operating without prior map knowledge. In exploration tasks, UDAQ outperformed frontier-based exploration across all environments, reducing travel distance by 36–44% and exploration time by 48–59%, while also substantially exceeding the performance of simpler reactive methods. Directional exploration experiments further demonstrated that oriented reward configurations can significantly improve exploration efficiency and room visitation order. These results confirm that UDAQ provides a robust, efficient, and flexible solution for autonomous navigation in unknown environments, unifying multiple navigation behaviors within a single reward-driven framework and operating entirely as a fully autonomous closed-loop system.