通过分析特征协方差结构，用轻量后处理方法抑制视觉语言模型幻觉。
Suppressing VLM Hallucinations with Spectral Representation Filtering
- 基于特征差异的协方差分解，识别出低秩幻觉模式。
- 在深层投影权重中施加软频谱滤波，降低幻觉率且不损失语义质量。
- 无需训练或修改架构，可通用适配多种视觉语言模型。
视觉语言模型（VLM）常因过度依赖语言先验和跨模态对齐不精确，生成图像中不存在的物体、属性或关系。本文提出一种轻量级、无需训练的后处理方法——频谱表示过滤（SRF），通过分析真实与幻觉描述特征差异的协方差结构，识别出低秩幻觉模式。利用特征空间中的结构化偏差，通过软频谱滤波器对深层vLLM层的前馈投影权重进行修正，均衡特征方差的同时保持语义保真度。相比解码或重训练方法，SRF完全后置操作，推理开销为零，无需架构改动。在三类VLM（LLaVA-1.5、MiniGPT-4、mPLUG-Owl2）上，于MSCOCO、POPE-VQA等基准测试中一致降低幻觉率，实现当前最优忠实性，且未损害生成质量。
原文摘要 · Abstract (English)
Vision-language models (VLMs) frequently produce hallucinations in the form of descriptions of objects, attributes, or relations that do not exist in the image due to over-reliance on language priors and imprecise cross-modal grounding. We introduce Spectral Representation Filtering (SRF), a lightweight, training-free method to suppress such hallucinations by analyzing and correcting the covariance structure of the model's representations. SRF identifies low-rank hallucination modes through eigendecomposition of the covariance of the differences between features collected for truthful and hallucinatory captions, revealing structured biases in the feature space. A soft spectral filter then attenuates these modes in the feed-forward projection weights of deeper vLLM layers, equalizing feature variance while preserving semantic fidelity. Unlike decoding or retraining-based approaches, SRF operates entirely post-hoc, incurs zero inference overhead, and requires no architectural modifications. Across three families of VLMs (LLaVA-1.5, MiniGPT-4, and mPLUG-Owl2), SRF consistently reduces hallucination rates on MSCOCO, POPE-VQA, and other visual tasks benchmarks, achieving state-of-the-art faithfulness without degrading caption quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。