不训练即可提升多模态模型定位准确率,减少幻觉。
Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attention

- 通过聚焦预测区域内的注意力计算,衡量预测置信度。
- 在多个模型和模态上,幻觉检测AUC提升7%至38%。
- 适用于图像、视频、音频定位,无需额外训练。
多模态大语言模型能生成局部化预测,如物体边界框或视频/音频事件的时间窗口,但常产生大量幻觉。模型自身的词元概率几乎无用:它混淆了定位质量与输入模糊性,且一旦模型确定,坐标词元几乎变为确定值。本文提出多词元局部注意力(MTLA):一种无需训练、后处理的评分方法,用于衡量预测词元对其声称区域的关注强度。先前基于注意力的检测器仅对整个模态求和并读取单个响应词元,是较弱的特例;我们证明,仅在声称区域内求和并聚合所有预测词元的注意力,可恢复更强的定位信号。该方法几乎可直接扩展至其他模态和任务:图像目标检测、视频与音频的时间定位。在多个MLLM家族和三种模态上,MTLA相比最佳现有无训练基线,幻觉检测AUROC提升7%至38%。作为重排序置信度分数使用时,其使一个开源80亿参数通用模型的零样本COCO检测AP从20.4提升至37.0,接近有监督检测器性能,且无需任何任务特定训练。
原文摘要 · Abstract (English)
Multimodal large language models can emit localized predictions, bounding boxes for objects and temporal windows for video and audio events, but they hallucinate these regions prolifically. The model's own token log-probabilities are nearly uninformative: they conflate grounding quality with input ambiguity, and coordinate tokens become near-deterministic once the model commits. We propose Multi-Token Localized Attention (MTLA): a training-free, post-hoc score that measures how strongly a prediction's tokens attend to the region they claim. Prior attention-based detectors, which sum attention over the entire input modality and read a single response token, are weaker special cases; we show that summing only within the claimed region and aggregating across all prediction tokens recovers a stronger grounding signal. The same recipe applies almost trivially to other modalities and tasks: object detection in images and temporal localization in video and audio. Across multiple MLLM families and three modalities, MTLA improves hallucination AUROC by +7 to +38 over the best prior training-free baseline. Used as a confidence score for re-ranking, it nearly doubles the zero-shot COCO detection AP of an open-source 8B generalist (from 20.4 to 37.0), narrowing the gap to supervised detectors without any task-specific training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。