通过文本引导融合多层视觉特征，有效减少大模型幻觉。
TGIF: Text-Guided Layer Fusion Mitigates Hallucination in Multimodal LLMs
- 用文本条件动态融合视觉编码器多层特征，提升视觉理解
- 在多个基准上显著降低幻觉率，同时保持或提升其他任务表现
- 无需修改视觉编码器，轻量模块适配主流多模态模型
多模态大语言模型通常仅依赖冻结视觉编码器的单个末层特征，未能充分利用其丰富的层次化视觉线索。现有模型仍存在严重视觉漂移幻觉，常依赖语言先验而非图像证据。尽管已有多种缓解策略聚焦于文本侧，但未改变视觉表征，也未挖掘视觉层间的丰富层次信息。现有跨层融合方法虽部分解决此问题，却采用固定融合权重，无法随查询变化。本文提出TGIF（Text-Guided Inter-layer Fusion），将编码器各层视为深度专家，根据输入提示动态预测视觉特征的融合方式。TGIF遵循直接外部融合原则，无需更新视觉编码器，额外开销极小。集成至LLaVA-1.5-7B后，在幻觉、OCR与VQA等任务中实现持续改进，同时在ScienceQA、GQA和MMBench上保持或提升性能。结果表明，查询感知、层次感知的融合是增强视觉对齐、减少幻觉的有效途径。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) typically rely on a single late-layer feature from a frozen vision encoder, leaving the encoder's rich hierarchy of visual cues under-utilized. MLLMs still suffer from visually ungrounded hallucinations, often relying on language priors rather than image evidence. While many prior mitigation strategies operate on the text side, they leave the visual representation unchanged and do not exploit the rich hierarchy of features encoded across vision layers. Existing multi-layer fusion methods partially address this limitation but remain static, applying the same layer mixture regardless of the query. In this work, we introduce TGIF (Text-Guided Inter-layer Fusion), a lightweight module that treats encoder layers as depth-wise "experts" and predicts a prompt-dependent fusion of visual features. TGIF follows the principle of direct external fusion, requires no vision-encoder updates, and adds minimal overhead. Integrated into LLaVA-1.5-7B, TGIF provides consistent improvements across hallucination, OCR, and VQA benchmarks, while preserving or improving performance on ScienceQA, GQA, and MMBench. These results suggest that query-conditioned, hierarchy-aware fusion is an effective way to strengthen visual grounding and reduce hallucination in modern MLLMs. Code: https://github.com/Linchenchen/TGIF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。