提出无需训练的视频模型纠错方法,缓解生成偏倚导致的幻觉问题。
Relaxing Anchor-Frame Dominance for Mitigating Hallucinations in Video Large Language Models

- 通过识别关键帧偏差,动态重分配解码器注意力,实现时间上更均衡的证据融合。
- 在多个视频大模型上测试,显著降低幻觉率,同时保持理解性能和推理速度。
- 适合关注视频生成可靠性、希望不改架构即可优化输出的开发者使用。
近期视频大语言模型(Video-LLMs)虽在视频理解方面表现强劲,但仍存在幻觉问题。现有缓解方法多依赖训练、输入修改或额外引导机制,却忽视了一个根本挑战:生成过程中模型过度依赖有限时间段的视觉证据,造成时间上不均衡的特征聚合。我们发现解码器阶段存在一种时间集中模式,将注意力质量最高的帧称为锚点帧。该偏差与输入视频无关,而是模型固有的结构或位置偏好,且其主导性与幻觉生成密切相关。为此,我们提出解码器侧时间重平衡(DTR),一种无需训练、可选层的推理阶段方法,在不改变视觉编码或引入辅助模型的前提下,对中后期解码层进行自适应注意力校准,减轻时间集中现象,提升被忽略帧的贡献度。实验表明,DTR在多种视频大模型家族中持续提升幻觉鲁棒性,同时保持优异的视频理解性能与高推理效率。
原文摘要 · Abstract (English)
Recent Video Large Language Models (Video-LLMs) have demonstrated strong capability in video understanding, yet they still suffer from hallucinations. Existing mitigation methods typically rely on training, input modification, auxiliary guidance, or additional decoding procedures, while largely overlooking a more fundamental challenge. During generation, Video-LLMs tend to over-rely on a limited portion of temporal evidence, leading to temporally imbalanced evidence aggregation across the video. To address this issue, we investigate a decoder-side phenomenon in which the model exhibits a temporally imbalanced concentration pattern. We term the frame with the highest aggregated frame-level attention mass the anchor frame. We find that this bias is largely independent of the input video and instead appears to reflect a persistent, model-specific structural or positional bias, whose over-dominance is closely associated with hallucination-prone generation. Motivated by this insight, we propose Decoder-side Temporal Rebalancing (DTR), a training-free, layer-selective inference method that rebalances temporal evidence allocation in middle-to-late decoder layers without altering visual encoding or requiring auxiliary models. DTR adaptively calibrates decoder-side visual attention to alleviate temporally imbalanced concentration and encourage under-attended frames to contribute more effectively to response generation. In this way, DTR guides the decoder to ground its outputs in temporally broader and more balanced video evidence. Extensive experiments on hallucination and video understanding benchmarks show that DTR consistently improves hallucination robustness across diverse Video-LLM families, while preserving competitive video understanding performance and high inference efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。