冻结的JEPA预测器可跨模型补全遮挡特征,提升识别准确率。
The JEPA Predictor: A Transferable Operator for Occluded Feature Completion
- 用冻结的JEPA预测器作为通用补全算子,通过线性投影适配不同主干模型。
- 在重遮挡下,对ImageNet-9和Stanford Dogs的准确率分别恢复至95%和52.1%(+36个百分点)。
- 无需重训练,适用于多类图像分类任务,尤其适合细粒度识别场景。
联合嵌入预测架构(JEPAs)在训练时联合学习编码器与预测器,但下游使用时仅保留编码器。预测器本质上是将可见上下文特征映射到遮挡位置特征的可迁移算子,正是部分视图分类所需。本文证明该算子可在不同编码器间通用。在高遮挡率下,保持冻结的JEPA预测器显著缩小与最强非-JEPA判别基线的性能差距。通过单一线性投影将I-JEPA与V-JEPA 2的预测器接入四个非-JEPA主干(CLIP、DINOv3、DINOv2、MAE),在500张ImageNet-1k图像上闭式求解。在ImageNet-9与Stanford Dogs数据集上,随遮挡比例K增加,每对主干-源模型的性能提升均单调上升。例如,CLIP搭配I-JEPA预测器,在重遮挡下于ImageNet-9恢复了遮挡损失的大部分精度;在Stanford Dogs上从15.9%提升至52.1%(+36个百分点)。机制可解析:投影在可见区域付出固定代价,预测器在遮挡区域提供递增收益,后者主导重遮挡情形。低遮挡率下投影代价超过收益,导致线性桥接失效。冻结的JEPA预测器作为跨模型的遮挡特征补全算子,无需重新训练任一模型,且每个遮挡比例仅需匹配线性探针即可实现。
原文摘要 · Abstract (English)
Joint-Embedding Predictive Architectures (JEPAs) train a predictor jointly with their encoder, but downstream deployment discards the predictor and reads features from the encoder alone. The predictor is, by construction, a learned operator from visible-context features to features at masked positions, the structure a partial-view classifier needs. We show that this operator is portable across encoder families. We first establish that, at heavy mask, retaining the frozen predictor on a JEPA encoder substantially closes the accuracy gap against the strongest non-JEPA discriminative baselines. We then bolt the frozen predictors of I-JEPA and V-JEPA 2 onto four non-JEPA hosts (CLIP, DINOv3, DINOv2, MAE) through a single linear projection between feature spaces, fit in closed form on 500 ImageNet-1k images. Across both ImageNet-9 and Stanford Dogs and across three mask fractions, the lift over each host's masked-encoder baseline grows monotonically with the mask fraction K in every host-donor pair. CLIP paired with the I-JEPA predictor recovers most of the accuracy that masking removed on ImageNet-9 at heavy occlusion, and lifts fine-grained Stanford Dogs from 15.9% to 52.1% (+36 pp). The mechanism is identifiable: the projection pays a fixed cost on visible patches and the predictor provides a growing benefit on masked patches; the benefit dominates the heavy-occlusion regime. At low K on fine-grained classification the projection cost exceeds the benefit, defining the boundary where the linear bridge breaks down. The frozen JEPA predictor functions as a portable operator for occluded feature completion across encoder families, requiring no retraining of either model while fitting matched linear probes per mask fraction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。