3D Self-Supervised Pre-Training and 3D-to-2D Transfer Learning for Brain Tumor MRI Classification

The labeled MRI datasets for brain tumors are generally very small, while there are a lot of unlabeled 3D scans. We introduce an in-domain, cross-dimensional self-supervised pipeline: we first train a 3D masked autoencoder (MAE) on multimodal BraTS2020 volumes and then deterministically transfer the trained 3D encoder to 2D slice classification through weight deflation (depth-averaging patch-embedding filters while directly transforming Transformer weights). We then fine-tune the resulting 2D ViT on two slice-level benchmarks (Kaggle four-class; Figshare/Cheng three-class).On the Kaggle dataset, the deflated initialization gives the most advantage during the low-label regime: at 10% labels it boosts test macro-F1 by 0.125 over DINO and 0.120 over random initialization, and it gets macro-F1 ≥0.80 with 25% labels (while DINO and random reach ≥0.80 at 50%). In the course of 3D pretraining, the optimal masking ratio is governed by the scale of unlabeled data: in the case of 50 BraTS instances, r =0.75 results in the lowest validation loss (0.0318), whereas in the case of 100– 200 instances the optimum shifts towards r =0.50 (best 0.0207 at 200), coming close to the full-run best (0.0201). On Cheng, 5-fold cross-validation demonstrates steady improvements over random initialization (macro-F1 0.8171±0.0128 vs. 0.7857± 0.0117). All in all, slightly performed 3D MAE pretraining on BraTS produces a domain-aligned, label-efficient initializer for 2D brain tumor classification, cutting down on the dependency on ImageNet weights.