FingerCap: Fine-Grained Finger-Level Hand Motion Captioning

Understanding human hand motion requires reasoning that goes beyond coarse action semantics to capture detailed finger articulation and interaction dynamics. However, existing Video-MLLMs primarily capture global action intent and often fail to represent fine-grained finger-level movements and contact semantics. In this work, we propose Fine-grained Finger-level Hand Motion Captioning (FingerCap), which aims to generate textual descriptions that capture detailed finger-level semantics of hand actions. To support this task, we curate FingerCap-40K, a large-scale corpus of 40K paired hand-motion videos and captions spanning two complementary sources: concise instruction-style finger motions and diverse, naturalistic hand–object interactions. Furthermore, we design HandJudge, an LLM-based rubric that measures finger-level correctness and motion completeness for effective evaluation. To establish a strong yet lightweight baseline under the FingerCap task, we introduce FiGOP (Finger Group-of-Pictures), a module tailored for Video-MLLM settings with sparse RGB sampling. FiGOP augments keyframes with intermediate 2D hand keypoints, enabling complementary motion cues without increasing the density of RGB frames. Experiments on FingerCap-40K show that strong open- and closed-source Video-MLLMs still struggle with finger-level reasoning, while our FiGOP-enhanced model yields consistent gains under both HandJudge and human studies. The dataset and benchmark are available at .