无需微调，让大模型精准定位多个图像区域。
Referring Multiple Regions with Large Multimodal Models via Contextual Latent Steering
- 通过预计算上下文向量引导模型理解多区域指代
- 在多个数据集上超越专用指代模型表现
- 适用于无需训练的通用大模型，适合快速部署
大视觉语言模型虽具备整体视觉理解能力，但在基于视觉提示的区域级感知任务中表现不足，尤其在同时指代多个区域或依赖全局上下文时。本文提出无需训练的上下文潜在引导方法（CSteer），通过预计算隐含区域区分与全局关注行为的上下文向量，在推理阶段动态编辑表示，实现对多区域的上下文感知指代。实验表明，采用CSteer的通用大模型在多个数据集上优于专门设计的指代模型，成为该领域新的最先进方法。代码已开源。
原文摘要 · Abstract (English)
Large Multimodal Models (LMMs) have recently demonstrated their proficiency in holistic visual comprehension. However, most of them struggle to tackle region-level perception guided by visual prompts, especially for cases where multiple regions are referred simultaneously, or scenarios where global contexts are necessary for precise visual referring. We introduce Contextual Latent Steering (CSteer), a training-free approach for guiding general LMMs to refer multiple regions contextually, without expensive fine-tuning or architectural modifications. CSteer starts with pre-computing contextual vectors that implicitly represent visual referring behaviors, such as differentiation among regions and attention to global contexts, followed by representation editing during inference time. Experimental results on multiple datasets indicate that general LMMs with CSteer outperform tailored referring LMMs in most cases, suggesting a promising solution in training-free, and setting new state-of-the-art for this field. Code is available at https://github.com/xing0047/csteer.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。