Automatic piano performance correction using a cross-modal visual network integrated with optical music recognition

Piano education traditionally relies on expert instructors to provide real-time feedback on performance errors, yet this model faces scalability challenges due to limited instructor availability and high costs. This paper presents a cross-modal visual network integrated with optical music recognition for automatic piano performance error detection and correction. The system employs a dual-stream architecture with Vision Transformer-based encoders that simultaneously process hand motion patterns and keyboard state changes through bidirectional attention mechanisms, enabling comprehensive error detection across pitch, rhythm, dynamic, and technical dimensions. Unlike audio-only approaches that detect errors after sound production, our method leverages visual cues to predict intended actions and identify errors before or during key presses. Experimental validation on the MUSCIMA++ dataset and our collected Piano Performance Error Dataset (PPED) containing 850 recordings from 45 pianists demonstrates that the system achieves 96.8% symbol recognition rate and 92.5% F1-score in error detection, substantially outperforming CNN-based methods (72.4%) and CRNN architectures (83.6%). Ablation studies confirm that cross-modal fusion contributes 6.8 percentage points improvement over single-modality approaches. The system maintains robust performance across difficulty levels from beginner (F1: 95.8%) to advanced repertoire (F1: 88.3%), providing interpretable visualizations of hand tracking and attention mechanisms that enable actionable pedagogical feedback for scalable music education.