通过分层抑制与融合，显著减少多模态大模型的幻觉生成。
LISA: A Layer-wise Integration and Suppression Approach for Hallucination Mitigation in Multimodal Large Language Models
- 分层设计：浅层保留视觉锚点，深层抑制错误信号。
- 在CHAIR_I上降低53.6%幻觉，在POPE上提升5.1%准确率。
- 即插即用，适配Qwen2.5-VL等主流多模态模型。
多模态大语言模型（MLLM）在图像描述等任务中表现优异，但易产生物体幻觉，即描述图像中不存在的物体。为缓解此问题，本文提出分层整合与抑制方法（LISA）。LISA利用MLLM各层的功能差异：浅层提供视觉锚定，中层编码语义，深层易放大虚假信号。首先，通过分层谱调制抑制深层过度放大的激活，同时保留浅层对齐线索；其次，采用基于锚点的路由机制，对选定层的词元级逻辑值进行软融合，实现解码过程中的自适应整合。LISA完全即插即用，可无缝集成至现有MLLM，包括Qwen2.5-VL。在多个基准测试中，其在CHAIR_I上将幻觉减少53.6%，在POPE上提升F1达5.1%，展现出强泛化能力。代码已开源。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) excel in vision-language tasks such as image captioning but remain prone to object hallucinations, where they describe objects that do not appear in the image. To mitigate this, we propose LISA, a Layer-wise Integration and Suppression Approach. LISA leverages the layer-wise functional roles in MLLMs: shallow layers provide visual grounding, middle layers encode semantics, and deep layers tend to amplify spurious signals. First, layer-wise spectral modulation stabilizes attention by suppressing over-amplified activations in deeper layers while preserving alignment cues in earlier layers. Second, token-level logits from selected layers are fused via anchor-based routing, with token-wise anchor selection and soft logit fusion enabling adaptive integration during decoding. LISA is fully plug-and-play and can be seamlessly integrated into existing MLLMs, including Qwen2.5-VL. Experiments on multiple benchmarks show that LISA reduces hallucinations by up to 53.6% in $\text{CHAIR}_\text{I}$ and improves POPE F1 by up to 5.1%, demonstrating strong generalization across models and tasks. Our code is available at https://github.com/zhlisa1010-eng/LISA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。