通过稀疏干预恢复被抑制的视觉信息，有效减少大模型幻觉
Revis: Sparse Latent Steering to Mitigate Object Hallucination in Large Vision-Language Models
- 基于潜在空间正交投影，精准定位并激活被压制的视觉信号
- 在标准基准上将物体幻觉率降低约19%，计算开销极低
- 适合关注视觉推理准确性与模型可解释性的研究者
尽管大型视觉语言模型具备强大能力，但仍频繁出现物体幻觉问题。原因之一是深层网络中视觉特征与预训练文本表示相互纠缠。为此，我们提出REVIS，一种无需训练的框架，旨在显式重激活被抑制的视觉信息。基于潜在空间几何结构，REVIS通过正交投影提取纯净的视觉信息向量，并采用校准策略，在发生抑制的精确深度进行稀疏干预。该手术式方法以极小计算成本有效恢复视觉信息。在标准基准上的实证评估表明，相较于现有最优基线，REVIS将物体幻觉率降低约19%，同时保持了模型的通用推理能力。
原文摘要 · Abstract (English)
Despite the advanced capabilities of Large Vision-Language Models (LVLMs), they frequently suffer from object hallucination. One reason is that visual features and pretrained textual representations often become intertwined in the deeper network layers. To address this, we propose REVIS, a training-free framework designed to explicitly re-activate this suppressed visual information. Rooted in latent space geometry, REVIS extracts the pure visual information vector via orthogonal projection and employs a calibrated strategy to perform sparse intervention only at the precise depth where suppression occurs. This surgical approach effectively restores visual information with minimal computational cost. Empirical evaluations on standard benchmarks demonstrate that REVIS reduces object hallucination rates by approximately 19% compared to state-of-the-art baselines, while preserving general reasoning capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。