Concepts from Representations: Post-hoc Concept Bottleneck Models via Sparse Decomposition of Visual Representations

Deep learning have achieved remarkable success in image recognition, yet their inherent opacity poses challenges for deployment in critical domains. Concept-based interpretations aim to address this by explaining model reasoning through human-understandable concepts. However, existing post-hoc methods and ante-hoc concept bottleneck models (CBMs), suffer from limitations such as unreliable concept relevance, non-visual or labor-intensive concept definitions, and model or data-agnostic assumptions. This paper introduces $\textbf{P}$ost-hoc $\textbf{C}$oncept $\textbf{B}$ottleneck $\textbf{M}$odel via $\textbf{Re}$presentation $\textbf{D}$ecomposition ($\textbf{PCBM-ReD}$), a novel pipeline that synergizes the strengths of both paradigms. PCBM-ReD automatically extracts visual concepts from a pre-trained encoder, employs multimodal large language models (MLLMs) to label and filter concepts based on visual identifiability and task relevance, and selects an independent subset via reconstruction-guided optimization. Leveraging CLIP’s visual-text alignment, it decomposes image representations into linear combination of concept text embeddings to fit into the CBMs abstraction. Our extensive experiments across 11 image classification tasks demonstrate that PCBM-ReD achieves state-of-the-art accuracy, narrows the performance gap with end-to-end models, and exhibits better interpretability. The code will be released.