通过上下文偏好调控，无需训练即可有效减少多模态大模型的幻觉问题。
Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigation
- 基于冲突样本提取上下文偏好向量，通过残差注入控制模型依赖
- 在多个模型上验证可显著降低物体幻觉率，且不增加推理延迟
- 适合需要快速部署、避免幻觉的视觉问答与内容生成场景
物体幻觉仍是多模态大语言模型（MLLMs）可靠部署的主要障碍。当前的推理阶段缓解方法主要假设幻觉源于视觉信息忽视，通过增强视觉依赖来纠正。然而，我们在多个MLLM上的系统性干预表明，过度强调视觉依赖反而可能加剧某些模型的幻觉，而适度减弱则有助于缓解。这说明将幻觉单纯归因于视觉不足是不充分的。我们认为图像作为上下文，同时与模型参数化知识和文本上下文竞争。为此，我们提出一种无需训练的框架——上下文偏好激活调控（CAS）。该方法通过两组设计好的冲突样本提取两个语义不同的上下文偏好向量（CPVs），并在推理时通过单次有符号残差注入的方式，作用于中早期MLP层，以控制信息依赖。实验表明，CAS能显著缓解物体幻觉，且不增加解码延迟，同时保持原生文本生成质量。
原文摘要 · Abstract (English)
Object hallucination remains a primary obstacle to the reliable deployment of Multimodal Large Language Models (MLLMs). Current inference-time mitigation methods mainly assume hallucinations stem from visual neglect, steering models to enhance visual reliance. In contrast, our systematic interventions on multiple MLLMs show that pushing toward more visual reliance may exacerbate hallucinations on some models, while less may mitigate hallucinations. This result suggests that attributing hallucinations solely to visual insufficiency is underdetermined. We argue that the image, as a context, simultaneously competes with the model's parametric knowledge and the textual context. For this, we propose a training-free framework, Context-Preference Activation Steering (CAS). It extracts two semantically distinct Context Preference Vectors (CPVs) via two small sets of designed conflict samples and applies them via single-pass signed residual injection at mid-early MLP layers during inference to control information reliance. Experiments show that CAS substantially mitigates object hallucinations without increasing decoding latency and preserves native text-generation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。