Consensus-JEPA: Learned Adaptive Targets from Multi-Mask Predictive Consistency

Title Consensus-JEPA: Learned Adaptive Targets from Multi-Mask Predictive Consistency Authors Frédéric David Blum (freddavidblum@catalystais.com) Catalyst AI — https://catalystais.com/ ORCID: 0009-0009-2487-2974 Upload type Publication → Preprint Publication date 2026-03-12 License All Rights Reserved (This retains full copyright. You can always relax to CC-BY-NC-ND later, but you cannot restrict after publishing under an open license.) Description (copy-paste this into the Zenodo description field) This note proposes Consensus-JEPA, a minimal extension of the Joint-Embedding Predictive Architecture (I-JEPA) in which the exponential-moving-average (EMA) target encoder is replaced by a learned consensus target module that aggregates predictions across multiple masking configurations. The central question in JEPA design—as of early 2026—is no longer whether to keep the EMA target encoder, but what kind of anchor best replaces it. Existing alternatives include frozen teachers (SALT; Li et al., 2025), pure distributional regularization without any teacher (LeJEPA; Balestriero & LeCun, 2025), and fixed external anchors (GMM-Anchored JEPA, 2026). Consensus-JEPA proposes a different answer: the target should be defined by what remains predictively stable across multiple partial observations of the same input, as judged by an adaptive, learned module. Key contributions: Leave-one-out consensus target in prediction space. For each spatial block and each held-out view, the target is computed by aggregating predictions from all remaining masking views. The consensus module operates entirely in prediction space—it never sees raw pixels. Per-block adaptive confidence with pairwise compatibility scoring. Each view's contribution to the consensus is weighted by (a) a local context-quality score based on compact mask descriptors (visible ratio, spatial distance to context, hole size, connectivity), and (b) a symmetric pairwise compatibility score that verifies mutual consistency between predictions from different views. Symmetry is enforced by construction through absolute-difference and sum inputs. A diversity regularizer with an entropy floor prevents single-view dominance while allowing justified concentration of weight when one view has a genuinely superior context. Consensus resilience diagnostic. A novel diagnostic (meaningful for K ≥ 4) measures whether the consensus survives partial ablation of contributing views, quantifying how distributed vs. concentrated the consensus is. Integration with SIGReg. Collapse prevention uses Sketched Isotropic Gaussian Regularization (Balestriero & LeCun, 2025), replacing VICReg-style heuristics with a provably optimal, single-hyperparameter distributional regularizer. Cross-scale extension. As a secondary contribution, the consensus is extended across spatial scales, requiring target stability under both fine and coarse masking. The proposal includes a complete specification of modules, exact inputs, training objective with block-aligned cross-view losses, a minimal training algorithm, spectral diagnostics (RankMe, mask-sensitivity index, spectral gap, consensus resilience), and two falsifiable predictions centered on robustness to mask geometry. To the best of our knowledge, among JEPA papers available on arXiv and OpenReview through March 12, 2026, we found no prior method that learns a prediction-space target by leave-one-out aggregation of multiple masked views with per-block confidence weighting and pairwise compatibility scoring. Status: Architectural hypothesis with falsifiable experimental protocol. Not yet empirically validated. Related work discussed: I-JEPA (Assran et al., 2023), V-JEPA 2 (Assran et al., 2025), LeJEPA (Balestriero & LeCun, 2025), SALT (Li et al., 2025), VL-JEPA (Chen et al., 2025), LLM-JEPA (Huang et al., 2025), GMM-Anchored JEPA (2026), Causal-JEPA (2026), seq-JEPA (2025), DSeq-JEPA (2025), JEPA as a Neural Tokenizer (2025). Keywords JEPA, self-supervised learning, consensus target, multi-view consistency, representation learning, collapse prevention, SIGReg, mask geometry, distribution shift robustness Related identifiers I-JEPA: https://arxiv.org/abs/2301.08243 (isSupplementedBy) LeJEPA: https://arxiv.org/abs/2511.08544 (isSupplementedBy) SALT: https://arxiv.org/abs/2509.24317 (isSupplementedBy) V-JEPA 2: https://arxiv.org/abs/2506.09985 (isSupplementedBy) Communities Machine Learning, Computer Vision, Self-Supervised Learning