Learning Mixture of Spatio-Temporal Experts for 3D Human Pose Estimation

The inherent depth ambiguity in monocular images or videos makes 3D human pose estimation an ill-posed problem, particularly in challenging scenarios such as occlusion and motion blur. To alleviate this problem, we propose MoEPose, the first Mixture-of-Experts-based framework for 3D human pose estimation. Specifically, we first design an Adjacency Relation Generator that treats each learnable adjacency matrix as an expert. This module leverages a gating network to compute a weight distribution over motion patterns for the input pose features, which is then used to adaptively fuse multiple experts, thereby generating an adjacency matrix that captures the relationships between nodes. Applied in parallel to spatial and temporal domains, the generator achieves cross-dimensional information interaction between nodes through linear transformations, thereby improving the capture of structural and temporal dependencies. Next, we further propose a Directional Consistency Loss to capture directional information. By aligning the predicted directions of virtual edges with their true directions, this loss effectively mitigates depth ambiguity. Extensive experiments demonstrate that our MoEPose achieves state-of-the-art performance on the Human3.6M and MPI-INF-3DHP datasets.