Demystifying GNN-to-MLP Knowledge Transfer: Theoretical Grounding and Dual-Stream Distillation Method

Graph Neural Networks (GNNs) have demonstrated remarkable effectiveness across various applications, yet their computational complexity poses significant scalability challenges. In contrast, structure-agnostic Multi-Layer Perceptrons (MLPs) offer computational efficiency and scalability but traditionally struggle with explicit graph data. To leverage the strengths of both, GNN-to-MLP Knowledge Distillation (KD) methods transfer relational inductive biases from GNNs to MLPs, equipping MLPs with graph-aware capabilities rivaling or even surpassing their teacher GNNs. In this paper, we theoretically answer how knowledge distillation unlocks MLPs’ potential for graph tasks from the perspective of training dynamics, demonstrating that label alignment in KD fundamentally reshapes the Neural Tangent Kernel (NTK) matrix of student MLPs to enable them to learn the teacher model's implicit graph bias. We further investigate finer-grained distillation paradigms, and reveal that conventional layer-wise output alignment fails to effectively align deep-layer graph propagation outcomes. To address this, we propose Dual-Stream Aligned MLP (DA-MLP), which incorporates complementary graph filters in a dual-stream architecture to simultaneously enhance feature space dimensionality for improved represenation alignment while preserving graph signals across different frequency bands. Comprehensive experiments on seven benchmark datasets validate that DA-MLP can be seamlessly integrated into existing knowledge distillation frameworks and consistently demonstrates performance enhancements in both transductive and inductive settings.