把指令嵌入图像，能减少视觉语言模型幻觉
Cure or Poison? Embedding Instructions Visually Alters Hallucination in Vision-Language Models
- 将文本指令直接嵌入图片，让模型只通过视觉通道理解内容
- 使Qwen2.5-VL的准确率提升4.1个百分点，幻觉率下降
- 适合关注多模态对齐与幻觉控制的研究者
视觉语言模型常因多模态信息对齐困难而产生幻觉。本文提出Prompt-in-Image方法，将文本指令直接嵌入图像，消除对独立文本输入的需求，强制模型通过视觉通道处理所有信息。在Qwen2.5-VL、LLaVA-1.5和InstructBLIP三个开源VLM上评估发现：该方法使Qwen2.5-VL的POPE准确率从80.2%提升至84.3%（+4.1个百分点），并降低MS-COCO上的幻觉率；但LLaVA-1.5和InstructBLIP性能严重下滑，准确率从约84%降至接近随机水平。分析表明，基于CLIP的编码器在两者中对嵌入文本区域存在过度注意力偏倚，破坏视觉理解；而Qwen的视觉编码器能稳健处理带文本的图像。关键在于，Prompt-in-Image缩小了Qwen的模态差距，通过单一模态统一信息处理，增强跨模态对齐。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) often suffer from hallucination, partly due to challenges in aligning multimodal information. We propose Prompt-in-Image, a simple method that embeds textual instructions directly into images. This removes the need for separate text inputs and forces the model to process all content through the visual channel. We evaluate this method on three popular open-source VLMs: Qwen2.5-VL, LLaVA-1.5, and InstructBLIP. The results reveal sharp differences. Prompt-in-Image improves Qwen2.5-VL's performance, increasing POPE accuracy by 4.1 percent (from 80.2 percent to 84.3 percent) and also reducing hallucination rates on MS-COCO. In contrast, LLaVA-1.5 and InstructBLIP experience a severe performance drop, with accuracy falling from around 84 percent to near-random levels. Through detailed analysis, we found that CLIP-based encoders in LLaVA and InstructBLIP exhibit excessive attention bias toward embedded text regions, disrupting visual understanding. In contrast, Qwen's vision encoder handles text-embedded images robustly. Crucially, Prompt-in-Image reduces Qwen's modality gap, enhancing cross-modal alignment by unifying information processing through a single modality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。