arXiv:2608.28216cs.CV2026-08

用一张参考图和描述,低成本实现复杂场景中目标的精准定位与缺失检测。

WALDO: One-Shot Exemplar-Conditioned Object Detection in Cluttered Scenes

论文配图:WALDO: One-Shot Exemplar-Conditioned Object Detection in Cluttered Scenes
图 1 · 摘自论文原文
  • 基于冻结的V-JEPA特征,仅用340万参数构建检测头,无需微调主干网络。
  • 在35个杂乱场景中达到0.461的AP@50,显著优于基线模型。
  • 适合对轻量级、零样本目标检测有需求的研究者或应用开发人员。

在仅提供一张参考图像和简短描述的前提下,从杂乱场景中定位特定目标实例并判断其是否存在,是大视觉语言模型常解决的任务。本文探究是否能以更低成本实现该能力——利用世界模型预训练所学的表征。提出WALDO:一个拥有340万可训练参数的一次性示例与语言条件检测头,读取冻结的V-JEPA 2.1特征,联合预测目标位置与存在性,且不更新主干网络。由于示例监督稀缺,训练样本通过实例标注合成:从真实边界框中提取示例,并构造包含同类别干扰物但不含目标实例的缺失场景。该任务易被捷径破解:若仅依赖裁剪尺寸,检测头可在无示例参与的情况下达到0.9998的缺失检测AUROC,因此文中报告了对照实验以关闭此捷径。在35个独立测试的杂乱场景上,WALDO获得0.461的catalogue AP@50,优于使用相同评分器的提示式Grounding DINO基线(0.306)。将DINOv3替换为V-JEPA,在576令牌网格下导致同类别缺失检测AUROC从0.880降至0.726,实例AP@50从0.201降至0.141,表明性能提升源于预训练目标而非输入分辨率。然而,实例级Success@1仅为0.190,与0.190的类别随机基线持平:世界模型特征虽有助于定位精度和缺失检测,但难以迁移至实例身份识别。

原文摘要 · Abstract (English)

Locating a specific object instance in a cluttered scene using a single reference image and a short description, and reporting when that instance is absent, large vision-language models usually address this task. We ask whether the same capability is available far more cheaply, from representations already learned by a world-model pretraining objective. We present WALDO, a one-shot exemplar- and language-conditioned detection head with 3.4M trainable parameters that reads frozen V-JEPA 2.1 features to jointly predict object localization and target presence, with no gradient on the backbone. Because exemplar-conditioned supervision is scarce, we synthesize training episodes from instance annotations, mining exemplars from ground-truth boxes and constructing absence cases that exclude the referenced instance while leaving same-category distractors in view. This is easy to get wrong: in the obvious implementation, crop size alone predicts the label, and a head trained on it reaches 0.9998 absence AUROC without ever consulting the exemplar, and we report the negative controls that close the shortcut. On 35 held-out cluttered scenes, WALDO achieves a 0.461 catalogue AP@50, compared to 0.306 for a prompted Grounding DINO baseline under an identical scorer. Substituting DINOv3 for V-JEPA under a matched 576-token grid drops within-category absence AUROC from 0.880 to 0.726 and instance AP@50 from 0.201 to 0.141, isolating the pretraining objective rather than input resolution as the source of the gain. Instance-level Success@1, however, reaches only 0.190 against a 0.190 category-chance floor: world-model features transfer to localization precision and absence detection but not to instance identity.

目标检测零样本视觉语言世界模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。