arXiv:2605.14908cs.CV2026-05

通过输入级注意力调控,让大模型更准定位视频中的物体。

SteerSeg: Attention Steering for Reasoning Video Segmentation

论文配图:SteerSeg: Attention Steering for Reasoning Video Segmentation
图 1 · 摘自论文原文
  • 用可学习提示和思维链引导注意力,聚焦目标区域
  • 在多个数据集上显著提升定位准确率,优于基线方法
  • 轻量级设计,仅训练少量提示,保持预训练能力

视频推理分割需根据自然语言描述定位视频帧中的对象,常涉及空间推理与隐含指代。现有方法利用冻结的大视觉语言模型(LVLM)提取注意力图作为空间先验,实现无需训练的定位。但这些注意力图针对文本生成优化,难以精准定位,常导致模糊的定位信号。本文提出SteerSeg,一种轻量级框架,识别注意力错位为关键瓶颈,提出从源头通过输入级条件调节注意力。该方法结合可学习软提示与基于推理的思维链(CoT)提示:软提示重塑注意力分布以生成更集中的地图,而CoT属性通过引导注意力至正确实例,解决相似对象间的歧义。生成的注意力图转化为关键帧上的点提示以指导分割模型,候选轨迹则通过相关性评分排序选择。本方法仅冻结LVLM与分割模型参数,仅训练少量软提示,保留预训练推理能力的同时显著提升定位性能。尽管仅在Ref-YouTube-VOS上训练,其在多个多样基准上均表现优异,显著增强LVLM的空间定位能力。

原文摘要 · Abstract (English)

Video reasoning segmentation requires localizing objects across video frames from natural language expressions, often involving spatial reasoning and implicit references. Recent approaches leverage frozen large vision-language models (LVLMs) by extracting attention maps and using them as spatial priors for segmentation, enabling training-free grounding. However, these attention maps are optimized for text generation rather than spatial localization, often resulting in diffuse and ambiguous grounding signals. In this work, we introduce SteerSeg, a lightweight framework that identifies attention misalignment as the key bottleneck in attention-based grounding and proposes to steer attention at its source through input-level conditioning. SteerSeg combines learnable soft prompts with reasoning-guided Chain-of-Thought (CoT) prompting. The soft prompts reshape the attention distribution to produce more spatially concentrated maps, while CoT-derived attributes resolve ambiguity among similar objects by guiding attention toward the correct instance. The resulting attention maps are converted into point prompts across keyframes to guide a segmentation model, while candidate tracklets are ranked and selected using correlation-based scoring. Our approach freezes the LVLM and segmentation model parameters and learns only a small set of soft prompts, preserving the model's pretrained reasoning capabilities while significantly improving grounding. Despite being trained only on Ref-YouTube-VOS, SteerSeg generalizes well across diverse benchmarks, significantly improving the spatial grounding capability of LVLMs. Project page: https://steerseg.github.io

视频分割注意力调控视觉语言模型零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。