Adapting CLIP for 3D Human Pose Estimation

Two-stage 3D human pose estimation has garnered remarkable progress, whereas ill-posed issue still poses great deterioration to the performance. During the 2D-3D projection, comprehension of the human topology serves as a significant cue to remove the ambiguities. However, limited feature representation restricts the grasp of human knowledge. In this light, we advocate for enriching the feature representation via integrating information from different sources. Contrastive imagetext pretrained models like CLIP demonstrate the capability of generalized representation. Exploiting the strong representation power of these models improves the performance of massive downstream tasks. Inspired by this, we resort to CLIP and delve into the integration of multi-source information with the distillation technique to reduce the location error of the regression task. Firstly, we endeavor to adapt the relation knowledge from CLIP text encoder to the transformer block of joint relation modeling module. Embedding of external language knowledge enriches the feature with the tailored adjuster and conveys the relation information from a quite different aspect. Secondly, we employ the CLIP visual encoder as another source to attain the knowledge from the vision aspect.We transfer the discrete human joints into the sketch image where human skeleton is depicted on a white board. Afterwards, the sketch drawing serves as the input of CLIP visual encoder, leading to a full understanding of human anatomy. We inject this representation into 3D human pose estimation network with the distillation technique where a dual-path distillation module is advanced to retain multimodal information. Extensive experiments on Human3.6M and MPIINF- 3DHP showcase that proposed method achieves competitive results over the previous approaches and improves the location precision, revealing the efficacy of our approach.