arXiv:2412.00622cs.CVcs.AI2024-12ICCV被引 11

提出新提示方法,让视觉语言检测器在红外/深度图上保持零样本能力。

Visual Modality Prompt for Adapting Vision-Language Object Detectors

  • 设计编码器-解码器视觉提示,动态适配不同模态输入
  • 在红外与深度数据集上达到全微调性能,零样本能力不下降
  • 适合需要跨模态部署且保留泛化性的目标检测场景

当测试于红外和深度等不同模态时,目标检测器的零样本性能会下降。尽管已有研究探索图像转换技术来适应新模态,但这些方法仅限于单一模态,且仅适用于传统检测器。近期的视觉语言检测器(如 YOLO-World 和 Grounding DINO)展现出良好的零样本能力,但尚未被适配至其他视觉模态。传统微调方法会损害其零样本性能。现有视觉提示策略对每张图像使用相同线性变换,效果有限。为此,本文提出 ModPrompt,一种无需牺牲零样本能力即可适配视觉语言检测器至新模态的视觉提示策略。具体采用编码器-解码器结构,并融合推理友好的模态解耦残差提示,提升适应鲁棒性。实证结果表明,该方法在两个视觉语言检测器(YOLO-World、Grounding DINO)及挑战性红外数据集(LLVIP、FLIR)和深度数据集(NYUv2)上,性能接近全微调,同时保持原始零样本能力。代码已开源:https://github.com/heitorrapela/ModPrompt。

原文摘要 · Abstract (English)

The zero-shot performance of object detectors degrades when tested on different modalities, such as infrared and depth. While recent work has explored image translation techniques to adapt detectors to new modalities, these methods are limited to a single modality and apply only to traditional detectors. Recently, vision-language detectors, such as YOLO-World and Grounding DINO, have shown promising zero-shot capabilities, however, they have not yet been adapted for other visual modalities. Traditional fine-tuning approaches compromise the zero-shot capabilities of the detectors. The visual prompt strategies commonly used for classification with vision-language models apply the same linear prompt translation to each image, making them less effective. To address these limitations, we propose ModPrompt, a visual prompt strategy to adapt vision-language detectors to new modalities without degrading zero-shot performance. In particular, an encoder-decoder visual prompt strategy is proposed, further enhanced by the integration of inference-friendly modality prompt decoupled residual, facilitating a more robust adaptation. Empirical benchmarking results show our method for modality adaptation on two vision-language detectors, YOLO-World and Grounding DINO, and on challenging infrared (LLVIP, FLIR) and depth (NYUv2) datasets, achieving performance comparable to full fine-tuning while preserving the model's zero-shot capability. Code available at: https://github.com/heitorrapela/ModPrompt.

目标检测多模态视觉语言提示学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。