自监督模型在跨数据集评估中误判新颖性,实为领域偏移陷阱。
Asleep at the Wheel: JEPA's Limitations in Evaluating Novel Driving Data

- 用JEPA预测视频嵌入的误差来识别罕见片段
- 跨数据集测试时准确率高,同数据集测试时仅达随机水平
- 轻量监督探针可使精度翻倍,暴露自监督目标缺陷
现代自动驾驶车队录制的视频远超人工审查能力,亟需自动剪辑筛选机制以识别稀有且值得复核的片段,从而优化模型应对异常场景的能力。本文测试了一种无标签方法:通过自监督联合嵌入预测架构(JEPA)的预测误差作为“新颖性”评分。一个冻结的V-JEPA视频编码器搭配轻量预测头,用于重建被掩码的片段嵌入,难以预测的嵌入对应片段被标记为值得关注。在训练于一个数据集、测试于其他数据集的现实协议下,该方法表现优异。然而我们发现,这种看似成功实为领域偏移导致:在单数据集公平基准上,该机制退化至随机水平,与无需训练的基线相当。在同一冻结嵌入上进行轻量监督探测,平均精度几乎提升一倍,表明瓶颈在于自监督目标本身,而非表示能力。本文以此为案例揭示自监督学习评估中的问题:跨数据集协议可能无意间奖励领域分离而非真正的新颖性。
原文摘要 · Abstract (English)
Modern autonomous-driving fleets record far more video than human reviewers can inspect. This motivates the need for an automatic clip triage mechanism, to surface rare and review-worthy clips, so that driving models can be fine-tuned to better handle unideal circumstances. We test a label-free approach that scores clips by the prediction-error "novelty" of a self-supervised joint-embedding predictive architecture (JEPA); a frozen V-JEPA video encoder is paired with a lightweight predictor head to reconstruct masked clip embeddings, and clips whose embeddings are hard to predict are flagged as interesting. Evaluated under a realistic protocol that trains on one dataset and tests against footage from others, this approach appears highly effective. We show that this apparent success is actually a domain-shift consequence: on a fair benchmark drawn from a single dataset, this mechanism collapses to chance and is on par with simple no-training baselines. A lightly supervised probe on the same frozen embeddings results in almost double the average precision, indicating that the bottleneck is indeed the self-supervised objective, rather than the representation. We present this as a study for evaluating the effectiveness of self-supervised learning, where cross-dataset protocols can silently reward domain separation over novelty.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。