arXiv:2608.08167cs.CVcs.LG2026-08

不训练模型,用统计方法过滤视觉语言模型的幻觉输出。

Wiener Representation Filtering for VLM Hallucination Suppression

论文配图:Wiener Representation Filtering for VLM Hallucination Suppression
图 1 · 摘自论文原文
  • 在语言模型表示空间中,通过无梯度的统计校准估计幻觉成分。
  • 对4个主流模型测试,幻觉率下降30%以上,且保持生成流畅性。
  • 适用于图像、视频和多轮对话场景,通用性强,推理速度不变。

视觉语言模型(VLMs)在开放式描述和视觉问答中表现优异,但常会虚构图像中不存在的对象、属性或关系,即“幻觉”问题。本文提出一种无需训练、事后处理的表示空间滤波方法,仅需一次离线校准,在少量配对数据上通过前向传播与经验二阶统计量估算协方差结构,无需反向传播或微调。该方法将隐藏状态建模为真实与幻觉成分的叠加,推导出基于协方差的维纳型最优滤波器增益,并通过特征分解实现逐模式衰减,满足稳定性准则,对估计噪声具有连续响应。在推理时,仅对选定深层的前馈输出投影进行一次修正,模型运行不变速。在LLaVA-1.5、MiniGPT-4、Gemma3和mPLUG-Owl2上实验显示,对CHAIR、POPE和MME数据集的幻觉现象有持续抑制效果,同时保持语句流畅性和整体响应质量。进一步在TempCompass视频理解与离散扩散语言模型的具身对话任务中验证了方法泛化性,证明其在时序推理和多步去噪场景中同样有效降低幻觉。

原文摘要 · Abstract (English)

Vision-language models (VLMs) excel at open-ended captioning and visual QA but often describe objects, attributes, or relations absent from the image, a phenomenon known as object hallucination. We propose a {training-free, post-hoc representation editing technique} that operates in the representation space of the language backbone. The method performs a lightweight, one-time offline calibration on a modest paired dataset to estimate the required covariance structures, using only forward passes and empirical second-order statistics with no gradient updates or fine-tuning, after which the correction is absorbed directly into the model's existing weights. By modeling hidden states as a superposition of truthful and hallucination-associated components, we derive a Wiener-type estimator whose optimal gains are given in closed form from the covariances of paired truthful and hallucinated representations. An eigendecomposition yields mode-wise attenuation that respects a stability criterion, i.e., the filter responds continuously to estimation noise. The correction is applied once to the feed-forward output projections of selected deeper layers, at inference time, the model runs unchanged and at the same speed. Experiments on LLaVA-1.5, MiniGPT-4, Gemma3, and mPLUG-Owl2 demonstrate consistent reductions in object hallucination on CHAIR, POPE, and MME while maintaining caption fluency and overall response quality. We further demonstrate the generality of our approach on the TempCompass video understanding benchmark and on discrete diffusion language models for grounded dialogue, showing that representation filtering reduces hallucinations even in temporal video reasoning and multi-step, sequence-wide denoising settings.

幻觉抑制视觉语言模型后处理维纳滤波

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。