IGRL: Imitation-Guided Reinforcement Learning for Sample-Efficient and Stable Robotic Control

Reinforcement learning, which excels in simulation, often fails to transfer to real robot manipulation, where rewards are sparse and control is continuous, owing to exploration failure and low efficiency. We propose imitation-guided reinforcement learning (IGRL) that combines stable guidance from expert demonstrations with the adaptability of online reinforcement learning through simple two-point integration. At each step, the actions proposed by a reinforcement policy and an independently trained imitation policy are evaluated using a target Q-network, and an action with a larger Q value is executed. The same principle shapes the TD target, which reduces variance and mitigates early instability. This method avoids demonstration oversampling and auxiliary regularization, lowers the hyperparameter sensitivity, and permits distinct architectures for the imitation and reinforcement components to maximize representation learning. For pixel inputs, a deformable ViT encoder leverages the global context, and a tree-based optimal action search efficiently explores continuous action sequences. Empirically, the IGRL surpasses strong baselines across five manipulation tasks and walker and quadruped locomotion, with wider margins as the difficulty increases. For a real robot, IGRL attains 0.81 accuracy on pick and 0.74 on pick and place, with smaller degradation under harder settings, improving the sample efficiency, convergence speed, and reproducibility.