通过分层干预视频证据,有效减少视频大模型的时空幻觉。
STEAR: Layer-Aware Spatiotemporal Evidence Intervention for Hallucination Mitigation in Video Large Language Models
- 按解码层差异选择视觉证据,精准干预高风险步骤。
- 在多个基准上提升忠实度与时间一致性,幻觉率显著降低。
- 适合关注视频理解可靠性的研究者与开发者。
视频大语言模型(Video-LLMs)仍易产生时空幻觉,常生成无视觉依据的细节或错误的时间关系。现有方法将幻觉视为统一的解码失败,采用全局共享的修正规则。我们观察到不同解码层在视觉定位和后期语言组合中作用不同,表明干预必须分层进行。基于此,提出STEAD框架:识别高风险解码步骤,从对定位敏感的中间层选取条件化视觉证据。该证据用于双重目的:恢复中间层缺失的局部定位,构建时序扰动的补丁级反事实样本,以否定后期解码中的不一致推理。由此,STEAR在高效单编码推理框架下同时缓解空间与时间幻觉。在多个代表性视频大模型主干和挑战性基准上的实验表明,STEAR持续降低幻觉率,提升忠实度、时间一致性和鲁棒性。结果证实,可靠视频解码依赖于在正确层级施加精确证据干预,而非全局惩罚。代码见附录。
原文摘要 · Abstract (English)
Video Large Language Models (Video-LLMs) remain prone to spatiotemporal hallucinations, often generating visually unsupported details or incorrect temporal relations. Existing mitigation methods typically treat hallucination as a uniform decoding failure, applying globally shared correction rules. We instead observe that decoder layers contribute differently to visual grounding and later linguistic composition, indicating that intervention must be layer-aware. Based on this insight, we propose STEAR, a layer-aware spatiotemporal evidence intervention framework. STEAR identifies high-risk decoding steps and selects token-conditioned visual evidence from grounding-sensitive middle layers. It uses this shared evidence for two coupled purposes: restoring missing local grounding in middle layers, and constructing temporally perturbed patch-level counterfactuals to falsify inconsistent reasoning during late-layer decoding. Consequently, STEAR mitigates both spatial and temporal hallucinations within an efficient single-encode inference framework. Experiments across representative Video-LLM backbones and challenging benchmarks demonstrate that STEAR consistently reduces hallucinations while improving faithfulness, temporal consistency, and robustness. Our results confirm that reliable video decoding relies on intervening on precise evidence at the right layer, rather than enforcing a global penalty. The code is provided in the Supplementary Material.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。