无需训练即可实现视频语义分割，通过注意力融合提升精度。
Decomposed Attention Fusion in MLLMs for Training-Free Video Reasoning Segmentation
- 将视频分割转为问答任务，用滚动机制提取注意力图
- 通过对象-背景与帧间互补融合，降低噪声并增强目标区域响应
- 兼容SAM2提示，适合希望零训练部署的开发者
多模态大语言模型（MLLMs）通过关注与文本查询相关的视觉标记来实现强大的视频理解。为直接适配无训练方式的定位任务，本文将视频推理分割建模为视频问答任务，并通过滚动机制提取注意力图。然而，原始注意力图存在噪声且与物体区域对齐不佳。为此，提出分解注意力融合（DecAF）方法，包含两个机制：(1) 对比式对象-背景融合，(2) 互补视频帧融合。该方法抑制无关激活，强化目标聚焦信号，可直接将注意力图转换为粗略分割掩码。此外，引入注意力引导的SAM2提示以获取精细掩码。与现有需联合训练MLLM与SAM的方法不同，本方法完全无需重训练。DecAF在参考与推理视频分割基准上均优于无训练方法，性能接近有训练方法。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) demonstrate strong video understanding by attending to visual tokens relevant to textual queries. To directly adapt this for localization in a training-free manner, we cast video reasoning segmentation as a video QA task and extract attention maps via rollout mechanism. However, raw attention maps are noisy and poorly aligned with object regions. We propose Decomposed Attention Fusion (DecAF), which refines these maps through two mechanisms: (1) contrastive object-background fusion and (2) complementary video-frame fusion. This method suppresses irrelevant activations and enhances object-focused cues, enabling direct conversion of attention maps into coarse segmentation masks. In addition, we introduce attention-guided SAM2 prompting for obtaining fine-grained masks. Unlike existing methods that jointly train MLLMs with SAM, our method operates entirely without retraining. DecAF outperforms training-free methods and achieves performance comparable to training-based methods on both referring and reasoning VOS benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。