Balancing Speed and Accuracy: A Mouth-Eye 3D Keypoints Alignment Framework for Large Poses

In fields such as fatigue detection, facial mouth-eye visual information is vital. Existing methods for extracting mouth-eye state information based on Keypoints mainly focus on small 2D pose angles, and serious accuracy degradation or even failure occurs for large poses. In this paper, we propose a new 3D Keypoints Alignment framework focusing on the Mouth-Eye(3D-KAME), which strikes a balance between speed and accuracy. First, a set of three-dimensional deformable model (3DMM) parameters are efficiently regressed based on a lightweight backbone network, MobileNetV3, thereby improving processing speed. To address the poor reconstruction of fine eye expressions in 3DMMs, often caused by insufficient supervision signals, we introduce a sparse key point-guided spatial attention mechanism (KGSA) and a local reprojection loss of the mouth and eyes with pose awareness for the precise reconstruction of detailed eye expressions, which improves the accuracy. Additionally for the fatigue detection task, EAR loss is introduced to further accurately express the eye geometry relationship. The model runs at over 47.5 FPS on a single NVIDIA RTX 2080 Ti GPU in tracking mode for video streaming, and achieves an average accuracy of 95.6% for mouth-eye state recognition over a (±90°) range.