通过时空语义残差提升视频表征，有效减少大模型幻觉。
Enhancing Video Representations with Spatiotemporal-Semantic Residual to Mitigate Hallucinations in Video Large Multimodal Models
- 用轻量MLP网络在推理时优化视频表征残差。
- 在EventHallusion上降低40.69%幻觉率，MMVU上提升18.36%。
- 无需修改模型，单次前向传播，适合部署于实际系统。
尽管视频大模型在视频理解任务中表现优异，但仍存在幻觉问题。现有推理时干预方法通常在对比解码框架下修改视频，但其启发式设计改进有限且增加推理延迟。为此，我们提出ViSSRes，一种基于轻量MLP结构的推理时干预方法，通过对比随机游走刻画视频表征的时空一致性，并引入条件互信息将视频表征与模型语义理解关联。在保持模型主干冻结的前提下，ViSSRes学习视频表征的残差，并从时空与语义一致性双重角度进行优化。推理时仅需一次前向传播，不引入显著额外开销。实验表明，ViSSRes使LLaVA-NeXT-Video在EventHallusion上的幻觉率降低40.69%，在MMVU上（CoT设置下）视频理解性能提升18.36%，验证了其在缓解幻觉方面的有效性。
原文摘要 · Abstract (English)
Although Video Large Multimodal Models have achieved strong performance in video understanding, they still suffer from hallucination. Existing inference-time intervention methods usually modify videos under the contrastive decoding framework, but their heuristic designs bring limited improvements and increase inference latency. To address these issues, we propose ViSSRes, an inference-time intervention method that enhances video representations through a lightweight MLP-style network. Specifically, we use a contrastive random walk approach to characterize the spatiotemporal consistency of video representations, and introduce conditional mutual information to associate video representations with the model's semantic understanding. With the model backbone kept frozen, ViSSRes learns residuals for video representations and optimizes them from both spatiotemporal and semantic consistency perspectives. During inference, ViSSRes requires only a single forward pass and introduces no substantial additional inference cost. Experiments show that ViSSRes reduces the hallucination rate of LLaVA-NeXT-Video on EventHallusion by 40.69% and improves video understanding on MMVU by 18.36% under the CoT setting, demonstrating its effectiveness in mitigating hallucinations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。