From Deep to Shallow: Orthogonal Polynomial Activation Functions for Compact Neural Networks

Although deep learning has achieved remarkable success in numerous fields, the move toward deeper architectures raises concerns about interpretability, computational load, and environmental cost. Traditional networks rely on fixed, hand-crafted activation functions, potentially limiting modeling of complex nonlinearities in Data. To address these limitations, Learnable Orthogonal Polynomial Activation Functions (LOPAFs) is proposed, where each neuron uses a distinct activation expressed as an orthogonal polynomial expansion, closely reflecting biological neuron behavior. To avoid over-parameterization, structure-aware pruning removes negligible terms, producing compact, efficient, and more interpretable shallow networks that require fewer samples and iterations to converge. Evaluated across MLPs, CNNs, GCNs, LSTMs, and Transformers on 259 datasets of varying domain, size, dimensionality, and class-imbalance, LOPAF-based models consistently outperform ReLU-based deep architectures. They deliver accuracy gains up to 71.42%, occasionally doubling performance, while cutting parameters by 99% and reducing training time through shallower designs. LOPAFs expose interpretable structures aligned with data distributions, promoting a shift from deep to shallow learning for enhanced explainability and energy consumption without compromising performance. LOPAFs emerge as a scalable alternative to cope with data nonlinearities, paving the way for compact and adaptive architectures.