arXiv:2607.06726cs.CV2026-07

好初始化让视觉归因更精准,只需选关键区域即可。

A Good Initialization is All You Need for Faithful Visual Attribution

论文配图:A Good Initialization is All You Need for Faithful Visual Attribution
图 1 · 摘自论文原文
  • 提出两种前向方法,用粗粒度候选启动精细搜索,提升效率。
  • 在图像分类和多模态模型中实现最高归因准确率,修复率达96%。
  • 紧凑的证据掩码可直接用于修复,不依赖完整排序结果。

忠实视觉归因旨在识别支持模型预测的图像区域。基于搜索的扰动方法通过遮蔽区域并测量得分变化,主导了插入-删除归因前沿,但通常输出所有区域的完整排序。许多应用,尤其是多模态大模型(MLLM)的归因与修复,仅需一个紧凑的top-k证据掩码。本文研究此掩码优先归因问题。精确为k个区域的掩码具有组合复杂性:有效证据可能依赖于细粒度区域间的交互。粗粒度分组虽能稳定早期搜索,但会聚合冗余内容;单步评分则可能遗漏高价值组合。我们提出两种前向方法: extsc{CoPAIR} 利用相位间隙诊断构建粗粒度单区域/成对候选,作为完整排序搜索的预热; extsc{TRACE} 直接搜索固定基数的细粒度掩码,结合交叉熵采样、精英保留与分布更新,并提供有限预算恢复分析。所得证据集可作为紧凑归因掩码输出,或用于初始化贪婪或相位窗口算法以生成完整排序。在ImageNet分类任务中,使用CLIP ViT-L/14、CLIP RN101与ResNet-101,我们的初始化搜索方法在包含式前向调用计账下建立新的归因最前沿。在POPE与RePOPE数据集上,使用Qwen2.5-VL-3B-Instruct与LLaVA-v1.5-7B, extsc{TRACE}+Greedy取得最强搜索型MLLM归因结果。直接 extsc{TRACE}掩码在单点修复中分别达到94.44%与96.00%的修复率,表明紧凑证据掩码可作为可行动的归因输出,而非仅是完整排序的前缀。

原文摘要 · Abstract (English)

Faithful visual attribution identifies which image regions support a model prediction. Search-based perturbation methods lead the insertion--deletion faithfulness frontier by masking regions and measuring score changes, but they usually output a complete ordering of all regions. Many applications, especially MLLM attribution and repair, only need a compact top-\(k\) evidence mask. We study this mask-first attribution problem. An exactly \(k\)-region mask is combinatorial: useful evidence can depend on interactions among fine regions. Coarse grouping can stabilize early search but aggregates redundant content, whereas one-step scoring can miss high-value combinations. We introduce two forward-only methods. \textsc{CoPAIR} uses a PhaseWin--Greedy gap diagnosis to construct coarse singleton/pair candidates that warm-start full-ordering search. \textsc{TRACE} directly searches fixed-cardinality fine-region masks with cross-entropy sampling, elite retention, and distribution updates, with a finite-budget recovery analysis. The resulting evidence set can be returned as a compact attribution mask or used to initialize Greedy or PhaseWin when a complete ranking is required. Across ImageNet classification with CLIP ViT-L/14, CLIP RN101, and ResNet-101, our initialized search methods establish a new state-of-the-art frontier for faithful full-ordering attribution under inclusive forward-call accounting. On POPE and RePOPE with Qwen2.5-VL-3B-Instruct and LLaVA-v1.5-7B, \textsc{TRACE}+Greedy gives the strongest search-based MLLM attribution results. Direct \textsc{TRACE} masks further achieve single-point RePOPE repair rates of \(94.44\%\) and \(96.00\%\), showing that compact evidence masks can be actionable attribution outputs, not merely prefixes of full rankings.

视觉归因多模态初始化压缩掩码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。