通过视觉锚点抑制大模型幻觉，不增加延迟。
Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision Language Models
- 从预填充残差中提取视觉证据方向，动态注入解码过程。
- 在多个模型上平均降低24.4%的幻觉指标，保持96%以上吞吐。
- 适合追求低延迟高准确视觉生成的部署场景。
大型视觉语言模型（LVLM）通常将视觉输入作为语言解码器的前缀处理。随着模型自回归生成文本，初始视觉信息会逐渐“稀释”，导致模型过度依赖语言先验并产生幻觉。现有方法通过对比输出或迭代修正来缓解此问题，但带来高昂延迟。本文提出鲁棒证据方向引导解码调节框架（RUDDER），通过提取模型预填充阶段残差更新中的稳健视觉证据方向（CARD），并由自适应门控机制（Beta Gate）调节，动态注入解码过程以维持视觉锚定。在LLaVA-1.5（7B/13B）、Idefics2、InstructBLIP和Qwen2.5-VL上的实验表明，采用贪婪解码时，RUDDER平均使CHAIR_S下降24.4%，CHAIR_i下降23.6%，且在不同架构间具有良好扩展性，同时保持>96.0%的吞吐率。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) typically process visual inputs as a prefix to the language decoder. As the model autoregressively generates text, this initial visual information inevitably undergoes "dilution" leading the model to over-rely on language priors and hallucinate objects. Existing interventions attempt to correct this by contrasting logits or iteratively refining outputs, but they incur prohibitive latency costs. We propose Residual-Update Directed DEcoding Regulation (RUDDER), a framework that counters visual dilution by creating a persistent visual anchor. We extract a robust evidence direction (CARD) directly from the model's prefill residual updates, and inject it into the decoding process. This injection is modulated by an adaptive gate, the Beta Gate, which acts as a trust mechanism and ensures the visual reminder is applied only when necessary. Experiments on LLaVA-1.5 (7B/13B), Idefics2, InstructBLIP, and Qwen2.5-VL demonstrate that RUDDER consistently mitigates hallucination (with greedy decoding, RUDDER reduces CHAIR_S by an average of 24.4% and CHAIR_i by 23.6% relative) and scales effectively across architectures, all while maintaining >96.0% throughput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。