提升多模态模型视觉证据定位准确率,解决碎片化与上下文干扰问题。
Evidence Recomposition and Predictive Context Residualization for Visual Attribution in Multimodal Large Language Models
- 通过重组多视角证据减少定位碎片化
- 在COCO Caption上使F1-IoU从39.10提升至44.45
- 适合需要精准视觉归因的研究者使用
多模态大模型虽具备强大视觉语言能力,但其词元级视觉证据难以解析。现有基于logit-lens的归因方法将每个视觉词元隐藏状态投影到词汇空间解释生成词,但存在上下文混杂与独立解码不匹配问题,常导致归因图破碎,并受前序文本词元的自回归上下文信号干扰。本文提出ERCR框架,包含证据重组(ER)与预测上下文残差化(PCR):ER通过多视角不同词元-区域映射聚合目标证据,降低单一读出网格带来的碎片化;PCR利用基于RBO的排名相关性估计前序词元上下文图,并从ER结果中减去拟合成分以抑制干扰。在LLaVA、Qwen2-VL与InternVL系列模型上,于COCO Caption、GranDf与OpenPSG数据集上的实验表明,ERCR有效提升目标词元的视觉证据质量,在Qwen2-VL-2B上,COCO Caption任务的TAM F1-IoU从39.10提升至44.45,GranDf任务从30.83提升至37.20。整体上,ERCR为词元级视觉证据分析提供了一种实用优化方案。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have achieved strong vision-language performance, yet their token-level visual evidence remains difficult to inspect. Recent logit-lens attribution methods project each visual-token hidden state into the vocabulary space to explain generated words, but this token-wise readout introduces a mismatch: visual tokens are context-mixed by the model, while the attribution score is decoded independently at each token location. This often produces fragmented attribution maps and can be further affected by autoregressive context signals from preceding text tokens. We propose ERCR, an attribution framework built from Evidence Recomposition (ER) and Predictive Context Residualization (PCR). ER aggregates target evidence across multiple views with different token-to-region assignments, reducing attribution fragmentation caused by a single readout grid. PCR estimates a preceding-token context map with RBO-based rank relevance and subtracts its fitted component from the ER map to suppress context-token interference. Experiments on LLaVA, Qwen2-VL, and InternVL families across COCO Caption, GranDf, and OpenPSG show that ERCR improves visual evidence for target tokens and mitigates preceding-token context interference under the existing evaluation protocol. On Qwen2-VL-2B, ERCR improves TAM F1-IoU from 39.10 to 44.45 on COCO Caption and from 30.83 to 37.20 on GranDf. Overall, ERCR provides a practical refinement for token-level visual evidence inspection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。