Revisiting Weight Initialization for Transfer Learning on Tabular Data: The Feature-Adaptive Variance Initialization (FAVI) Approach

Transfer learning has advanced significantly in domains like computer vision and natural language processing, yet its application to tabular data remains challenging, with traditional models like XGBoost often outperforming deep learning approaches due to issues like variance instability and slow convergence. This study investigates the impact of weight initialization techniques on transfer learning efficacy during partial fine-tuning, hypothesizing that optimized methods enhance (i) variance stability (consistent activation and gradient magnitudes), (ii) convergence speed (faster loss reduction), and (iii) generalization (improved out-of-distribution accuracy). Established techniques including Xavier/Glorot, He/Kaiming, Orthogonal, and fan-averaged initializations are evaluated on modern foundation models like TabuLa-8B, using benchmarks such as OpenML and CC-18 (72 datasets). Additionally, Feature-Adaptive Variance Initialization (FAVI) is proposed, which adapts variances based on per-feature statistics, which is mathematically proven to preserve unit output variance. Empirical results demonstrate 15–20% improvements in convergence speed and 1.5–2.5% in generalization. Research contributions include a theoretical formulation of FAVI and insights advancing tabular data modeling.