通过放大视觉细节提升模型准确性，减少幻觉生成。
Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decoding
- 迭代识别关键视觉区域并放大，引导模型关注细节
- 显著降低幻觉率，同时保持强推理能力
- 无需微调，适合希望提升输出真实性的研究者
现有视觉语言模型常因视觉幻觉导致生成内容与图像输入不符。现有方法主要通过对比学习减少语言偏差或在解码时增强视觉嵌入权重来缓解问题，但对细粒度视觉信息捕捉能力有限。本文提出感知放大器（Perception Magnifier, PM），一种新型视觉解码方法：通过注意力机制迭代识别相关视觉标记，并放大对应区域，在每一步解码中既增强关键区域的感知，又保留整体结构与上下文信息。实验表明，PM不仅显著降低幻觉率，还能提升语言生成质量，同时维持强大的推理能力。代码已开源。
原文摘要 · Abstract (English)
Existing vision-language models (VLMs) often suffer from visual hallucination, where the generated responses contain inaccuracies that are not grounded in the visual input. Efforts to address this issue without model finetuning primarily mitigate hallucination by contrastively reducing language biases or amplifying the weights of visual embedding during decoding. However, these approaches remain limited in their ability to capture fine-grained visual details. In this work, we propose the Perception Magnifier (PM), a novel visual decoding method that iteratively isolates relevant visual tokens based on attention and magnifies the corresponding regions, spurring the model to concentrate on fine-grained visual details during decoding. By magnifying critical regions while preserving the structural and contextual information at each decoding step, PM allows the VLM to enhance its scrutiny of the visual input, hence producing more accurate and faithful responses. Extensive experimental results demonstrate that PM not only achieves superior hallucination mitigation but also enhances language generation while preserving strong reasoning capabilities. Code can be found at https://github.com/ShunqiM/PM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。