arXiv:2510.13808cs.CV2025-10被引 3

用轻量探针让视觉语言模型快速适应新场景,不丢旧能力。

VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models

  • 在视觉编码器加可学习探针,只微调少量参数。
  • 跨视角/模态/任务测试中均超越现有方法,性能提升显著。
  • 适合需快速适配新环境又不想重训的工程应用。

大型视觉语言模型(VLM)在通用视觉推理上表现优异,但在部署到与预训练数据分布差异大的新领域时性能严重下降。现有领域自适应方法依赖微调标准VLM组件,但更新不同组件会导致模型无法学习领域特定表征或遗忘已有能力。我们提出视觉上下文探针(VisCoP),一种参数高效自适应框架,在VLM视觉编码器中引入一组紧凑的可学习视觉探针。通过这些探针学习领域特定视觉表征,同时仅需对预训练模型组件进行极少更新,有效适应新领域且不损失原有知识。我们在三种挑战性自适应场景下评估:跨视角(外视角到第一人称视角)、跨模态(RGB到深度图)、跨任务(人类理解到机器人控制)。在所有场景中,VisCoP持续优于现有自适应策略,实现更优的目标域性能,同时保持源域上的预训练能力。结果表明,轻量视觉探针是应对显著分布偏移的有效且稳健解决方案。代码、模型与评估协议见 https://github.com/dominickrei/VisCoP。

原文摘要 · Abstract (English)

Large Vision Language Models (VLMs) excel at general visual reasoning but experience significant performance degradation when deployed in novel domains that exhibit substantial distribution shifts from their pretraining data. Existing domain adaptation methods rely on finetuning standard VLM components; however, depending on which components are updated, these approaches either limit the model's ability to learn domain-specific representations or cause catastrophic forgetting of previously acquired capabilities. We introduce Vision Contextualized Probing (VisCoP), a parameter-efficient adaptation framework that augments the VLM vision encoder with a compact set of learnable visual probes. By learning domain-specific visual representations through these probes while requiring only minimal updates to pretrained model components, VisCoP effectively adapts to new domains without sacrificing existing knowledge. We evaluate VisCoP across three challenging adaptation settings: cross-view (exocentric to egocentric), cross-modal (RGB to depth), and cross-task (human understanding to robot control). Across all scenarios, VisCoP consistently outperforms existing domain adaptation strategies, achieving superior target-domain performance while preserving the pretrained VLM's capabilities on the source domain. These results demonstrate that lightweight visual probing provides an effective and robust solution for adapting VLMs under substantial distribution shifts. Code, models, and evaluation protocols are available at https://github.com/dominickrei/VisCoP.

视觉语言模型领域自适应轻量级微调探针机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。