Asymmetric Distillation and Information Retention in Capacity-Constrained Cross-Modal Transfer
Abstract
Knowledge distillation between asymmetric architectures often induces severe geometric constraints on the learned representation space. In this work, we investigate the Dimensional Collapse phenomenon when distilling a 500M parameter global Vision Transformer (CLIP ViT-B/32) into strictly capacity-constrained, local-receptive-field CNNs (0.5M to 8.0M parameters)
Fields of Study & Subjects
Advanced Neural Network Applications
Domain Adaptation and Few-Shot Learning
Multimodal Machine Learning Applications
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 27 February 2026
- Language
- EN
- Document type
- Preprint