通过动态调节修正强度,有效减少视觉语言模型的幻觉问题。
Mitigating Hallucination in Vision-Language Models through Barrier-Regulated Adaptive Closed-form Steering

- 根据模型注意力变化实时检测视觉定位漂移,仅在需要时修正。
- 在不训练额外网络的情况下,将幻觉指标降低9.4分,提升2.7分。
- 无需重训模型,推理速度接近贪婪解码,适合部署在真实场景。
大型视觉语言模型在生成过程中常出现输入图像中不存在的物体幻觉,主要因为视觉定位能力随解码过程减弱。现有推理阶段缓解方法通过修改对数或隐藏状态来纠正,但存在三个关键缺陷:缺乏显式定位目标、在模型已正确时仍干预、修正强度固定无法适应严重程度。本文提出训练无依赖的BRACS(Barrier-Regulated Adaptive Closed-form Steering)框架,通过屏障调节的自适应闭式导向机制解决上述问题。BRACS监控模型自身注意力以衡量视觉定位,仅当定位能力下降时才对隐藏状态进行修正。修正更新以闭式解析形式计算,无需训练辅助网络或重训模型。在LLaVA-1.5-7B和Qwen-VL-Chat上的实验表明,BRACS在幻觉基准测试中持续优于先前方法,将CHAIR$_s$降低9.4点,提升POPE F1 2.7点,同时在四个通用多模态基准上保持或提升性能。该方法效率高,达到贪婪解码80%吞吐量,平均速度为基线的1.3倍。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) often hallucinate objects that are not present in the input image, largely because visual grounding weakens as decoding progresses. Existing inference-time mitigation methods modify logits or hidden states throughout generation, but they suffer from three key limitations: they lack an explicit grounding objective, intervene even when the model is already well-grounded, and use fixed correction strengths that do not adapt to the severity of grounding failure. We propose BRACS (Barrier-Regulated Adaptive Closed-form Steering), a training-free steering framework that addresses these issues through barrier-regulated adaptive closed-form steering. BRACS monitors the model's own attention to measure visual grounding and applies corrections to the hidden states only when grounding deteriorates. The corrective update is computed analytically in closed form, requiring no training of auxiliary networks or model retraining. Experiments on LLaVA-1.5-7B and Qwen-VL-Chat show that BRACS consistently outperforms prior methods on hallucination benchmarks, reducing CHAIR$_s$ by 9.4 points and improving POPE F1 by 2.7 points, while matching or improving performance on four general multimodal benchmarks. BRACS also remains efficient, operating at 80% of greedy decoding throughput and achieving 1.3 times higher speed on average than the baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。