Label-Kernel Correlation and an Improved Optimization Theory for Neural Networks

Neural Tangent Kernel (NTK) theory offers a principled framework for studying optimization and generalization in over-parameterized neural networks, under constant-kernel, yet it’s convergence theory predictions often present a big gap in the speed of learning observed in practice. This thesis investigates label-Kernel correlation as a unifying quantity that links target structure to kernel geometry and helps explain the theory-practice gap. We pursue two objectives: (i) characterize how label-kernel alignment improves optimization, and (ii) determine the training dynamics of alignment across architectures. On the theoretical side, we develop an improved optimization theory by understanding the interaction between target projections over the eigen directions and eigenvalues. On the experimental side, we track alignment and layer-wise alignment for MLPs, CNNs, and ResNet-18 under SGD-family and adaptive optimizers. We find consistent regimes where SGD-family promote stronger alignment growth in earlier layers with decreasing alignment growth toward later layers, while Adam exhibits more heterogeneous layer-wise patterns. To interpret these trends, we complement alignment with normalized per-layer gradient-of-loss norms and normalized per-layer weight-update energies, which frequently mirror the observed layer-wise alignment trends. Overall, the thesis provides improvement in existing optimization theory for neural networks using label-NTK correlation under constant NTK regime and analyzes the correlation under training in finite width neural networks