Linear Separability and Feature Importance Analysis in Large Language Model Representation Spaces

Large Language Models (LLMs) have achieved remarkable improvements in their capabilities, yet it remains unclear how these advances are reflected in their internal representation spaces. This paper conducts a systematic within-family empirical study on the relationship between model capability and representation structure across six models spanning three major LLM families (Qwen, LLaMA, and Mistral). Using standardized linear probing and trainonly feature attribution, we analyze layer-wise representation separability and dimensional sparsity on the MMLU-Pro dataset. Our results reveal three key findings. First, higher-performing models exhibit consistently higher linear separability (up to 87.3%), suggesting that capability improvements are associated with more linearly accessible representation structures. Second, task-relevant information is concentrated in extremely low-dimensional subspaces: just 2.8% of dimensions preserve 73–80% of classification performance, while 14% maintain over 92%. Third, different model families exhibit distinct sparsity signatures, with Qwen showing sharper concentration, LLaMA more uniform separability across layers, and Mistral more dispersed importance patterns. These findings provide empirical evidence on how model improvements shape latent spaces and have implications for interpretability, pruning, and subspace-based model compression.