AXON: Action Characterization Through Cross-Modal Knowledge Distillation for Neurodiverse Individuals

Understanding the communicative behaviors of non- and minimally-speaking individuals with autism spectrum disorder (ASD) and other complex neurodevelopmental disorders (NDDs) remains a critical challenge for both clinical support and machine learning research. However, developing automated systems for this task is hindered by data scarcity, privacy concerns, heterogeneous and idiosyncratic actions, and the significant domain shift from neurotypical to neurodiverse populations. To address these challenges, we first present a novel, large-scale, privacy-preserving 3D skeleton action recognition dataset with 2,721 samples capturing in-home interactions of nonverbal individuals with ASD and complex NDDs. Second, we propose AXON, a novel cross-modal knowledge distillation method that transfers the rich semantic understanding of a pre-trained CLIP model to a graph-based skeleton model, outperforming other cross-modal knowledge distillation baselines in classifying subtle communicative acts. We further introduce a gradient-based interpretability analysis method to characterize how individuals with ASD and complex NDDs perform communicative actions. Our analysis reveals both population- and individual-level communicative styles, showcasing individual biases and idiosyncratic movements. Our foundational study helps the development of more adaptive and personalized augmentative technologies, aiming to foster greater communicative autonomy and understanding for this underserved population.