用稀疏自编码器增强视觉信息，减少多模态模型幻觉
SAVE: Sparse Autoencoder-Driven Visual Information Enhancement for Mitigating Object Hallucination
- 通过稀疏自编码器定位视觉理解特征，引导模型聚焦真实图像内容
- 在CHAIR_S上提升10个百分点，多个基准测试均表现更优
- 无需训练，适配多种模型，适合追求鲁棒性的视觉生成研究者
尽管多模态大语言模型（MLLMs）已取得显著进展，但仍易受语言先验和视觉信息丢失影响而产生物体幻觉。为此，我们提出SAVE（稀疏自编码器驱动的视觉信息增强框架），通过引导模型沿稀疏自编码器（SAE）潜在特征前进来缓解幻觉问题。一个二分类物体存在性问答探针识别出最能反映模型视觉信息处理的SAE特征，称为视觉理解特征。沿这些特征引导模型可强化基于图像的视觉理解，有效降低幻觉。该方法设计简洁，在标准基准上超越现有无训练方法，在CHAIR_S上提升10%p，POPE与MMHal-Bench上持续增益。跨多个模型与层的评估验证了方法的鲁棒性与泛化能力。进一步分析表明，沿视觉理解特征引导可抑制不确定物体标记生成，并增强对图像标记的关注，从而减轻幻觉。代码已开源：https://github.com/wiarae/SAVE。
原文摘要 · Abstract (English)
Although Multimodal Large Language Models (MLLMs) have advanced substantially, they remain vulnerable to object hallucination caused by language priors and visual information loss. To address this, we propose SAVE (Sparse Autoencoder-Driven Visual Information Enhancement), a framework that mitigates hallucination by steering the model along Sparse Autoencoder (SAE) latent features. A binary object-presence question-answering probe identifies the SAE features most indicative of the model's visual information processing, referred to as visual understanding features. Steering the model along these identified features reinforces grounded visual understanding and effectively reduces hallucination. With its simple design, SAVE outperforms state-of-the-art training-free methods on standard benchmarks, achieving a 10\%p improvement in CHAIR\_S and consistent gains on POPE and MMHal-Bench. Extensive evaluations across multiple models and layers confirm the robustness and generalizability of our approach. Further analysis reveals that steering along visual understanding features suppresses the generation of uncertain object tokens and increases attention to image tokens, mitigating hallucination. Code is released at https://github.com/wiarae/SAVE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。