Orthogonally Regularized Autoencoder for Dual Structure Preservation in Glove Embeddings

Dimensionality reduction of word embeddings presents a fundamental trade-off: reconstruction fidelity versus semantic structure preservation. Through comprehensive benchmarking of autoencoder variants (Vanilla, Contractive, and Variational) against traditional methods (principal component analysis [PCA] and Uniform Manifold Approximation and Projection [UMAP]) on GloVe embeddings, a clear specialization pattern is revealed. Vanilla autoencoders achieve superior reconstruction (MSE: 0.0043), PCA excels at structure preservation (Continuity: 0.7176), and UMAP dominates clustering (Coherence: 0.3076). An Orthogonally Regularized Autoencoder is proposed that strategically balances these objectives, achieving 80% of PCA’s structural benefits with 50% better reconstruction. While no method dominates all metrics, this work provides principled guidance for method selection based on application requirements and challenges the assumption that complex regularization necessarily improves performance.