arXiv:2602.00621cs.CV2026-02被引 5

通过分析神经元激活机制,提升视觉语言模型的可解释性并减少幻觉。

Towards Interpretable Hallucination Analysis and Mitigation in LVLMs via Contrastive Neuron Steering

  • 用稀疏自编码器分解视觉特征,识别出图像特异和始终活跃的神经元。
  • 抑制错误激活的图像特异神经元,可有效降低幻觉率并提升视觉对齐度。
  • 适用于各类多模态任务,尤其适合需要高可信输出的场景。

LVLMs 在多模态理解与生成方面表现卓越,但仍易产生幻觉。现有方法多聚焦于输出层面调整,未深入探究内部机理。本文从表征层面出发,引入稀疏自编码器(SAEs)将密集视觉嵌入分解为稀疏可解释的神经元,发现图像特异神经元常因干扰或异常激活导致幻觉,而始终活跃神经元保持稳定。通过选择性增强或抑制图像特异神经元,可实现对输出的可控干预,提升视觉对齐并减少幻觉。基于此提出对比神经元调控(CNS),通过对比干净与噪声输入识别图像特异神经元,强化信息量高的神经元,抑制扰动引发的激活,生成更鲁棒、语义准确的视觉表示。该方法在预填充阶段运行,兼容现有解码阶段方法。在多组幻觉专项与通用多模态基准测试中,CNS持续降低幻觉率,同时保持整体多模态理解能力。

原文摘要 · Abstract (English)

LVLMs achieve remarkable multimodal understanding and generation but remain susceptible to hallucinations. Existing mitigation methods predominantly focus on output-level adjustments, leaving the internal mechanisms that give rise to these hallucinations largely unexplored. To gain a deeper understanding, we adopt a representation-level perspective by introducing sparse autoencoders (SAEs) to decompose dense visual embeddings into sparse, interpretable neurons. Through neuron-level analysis, we identify distinct neuron types, including always-on neurons and image-specific neurons. Our findings reveal that hallucinations often result from disruptions or spurious activations of image-specific neurons, while always-on neurons remain largely stable. Moreover, selectively enhancing or suppressing image-specific neurons enables controllable intervention in LVLM outputs, improving visual grounding and reducing hallucinations. Building on these insights, we propose Contrastive Neuron Steering (CNS), which identifies image-specific neurons via contrastive analysis between clean and noisy inputs. CNS selectively amplifies informative neurons while suppressing perturbation-induced activations, producing more robust and semantically grounded visual representations. This not only enhances visual understanding but also effectively mitigates hallucinations. By operating at the prefilling stage, CNS is fully compatible with existing decoding-stage methods. Extensive experiments on both hallucination-focused and general multimodal benchmarks demonstrate that CNS consistently reduces hallucinations while preserving overall multimodal understanding.

视觉语言模型幻觉抑制神经元分析可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。