Cross-Modal Matching of Lower Body Skeleton and Insole Pressure for Identity Recognition

In privacy-sensitive scenarios where traditional biometric cues such as faces or voices are unavailable, personal identification becomes a significant challenge. This work presents a cross-modal approach that combines lower-body skeletal data, captured by an RGB camera, with foot pressure distributions collected from smart insoles. The use of these non-intrusive sensors enables identity recognition without compromising user privacy. The two modalities are encoded into a unified three-channel image and processed using a deep neural architecture that integrates a pre-trained VGG16 and a long short-term memory (LSTM) network to learn cross-modal similarity. Identity matching is formulated as a bipartite graph problem, where similarity scores guide the pairing of anonymous skeletal data with ID-tagged insole readings. Experiments show enhanced performance, with pairing accuracy rising from 76.4% to 90.4%, and user identification rates doubling compared to a baseline KNN under long-duration monitoring.