gmum/beyond_cls: Initial release for "Beyond [cls]: Exploring the true potential of Masked Image Modeling representations" NCN 2023/49/N/ST6/03268

Masked Image Modeling (MIM) has emerged as a promising approach for Self-Supervised Learning (SSL) of visual representations. However, the out-of-the-box performance of MIMs is typically inferior to competing approaches. Most users cannot afford fine-tuning due to the need for large amounts of data, high GPU consumption, and specialized user knowledge. Therefore, the practical use of MIM representations is limited. In this paper we ask what is the reason for the poor out-of-the-box performance of MIMs. Is it due to weaker features produced by MIM models, or is it due to suboptimal usage? Through detailed analysis, we show that attention in MIMs is spread almost uniformly over many patches, leading to ineffective aggregation by the [cls] token. Based on this insight, we propose Selective aggregation to better capture the rich semantic information retained in patch tokens, which significantly improves the out-of-the-box performance of MIM. This research has been supported by the flagship project entitled “Artificial Intelligence Computing Center Core Facility” from the Priority Research Area DigiWorld underthe Strategic Programme Excellence Initiative at the Jagiellonian University. The research of Marcin Przewięźlikowski was supported by the National Science Centre (Poland),grant no. 2023/49/N/ST6/03268. The research of Wojciech Jasiński was supported by the National Science Centre (Poland), grant no. 2022/47/B/ST6/03397. The research of Marek Śmieja was supported by the National Science Centre (Poland), grant no. 2022/45/B/ST6/01117. The research of Bartosz Zieliński was supported by the National Science Centre (Poland), grant no. 2023/50/E/ST6/00469. We gratefully acknowledge Polish high-performance computing infrastructure PLGrid (HPC Center: ACK Cyfronet AGH) for providing computer facilities and support within computational grant no. PLG/2024/017148. We thank Marcin Sendera, Adam Pardyl, Turhan Can Kargin, Bill Psomas, Michał Pietruszka, and Klaudia Bałazy for fruitful discussions and feedback over the course of this work.