用虚构视频测试大模型真理解能力，发现其常忽略物理错误。
VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding
- 用生成视频设计违背物理的测试场景，检验模型真实理解力。
- 主流模型在标准评测表现好，却常漏掉明显物理错误。
- 微调后识别错误能力提升，且不影响原有性能，适合评估模型可靠性。
视觉语言模型（VLMs）在视频理解任务中表现强劲，但其是否真正理解视觉内容仍存疑——可能仅学习了视觉与语言间的浅层关联。真实世界理解需依赖物理和常识推理，而现有评测多使用与训练数据相似的真实视频，高分未必代表真理解。为此，我们提出基于负向控制的测试方法，采用包含物理不可能或逻辑矛盾事件的合成视频。我们构建了VideoHallu数据集，通过Veo2、Sora和Kling生成违反物理与常识的场景，并由专家标注四类问题的问答对。对Qwen-2.5-VL、Video-R1、VideoChat-R1等领先模型的测试表明，尽管在MVBench和MMVU等基准上表现优异，它们仍频繁忽略这些错误，暴露出视觉推理短板。在VideoHallu上进行强化学习微调可有效提升对这类错误的识别能力，且不损害原有基准性能。数据集已开源：https://github.com/zli12321/VideoHallu.git。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have achieved strong results in video understanding, yet a key question remains: do they truly comprehend visual content or only learn shallow correlations between vision and language? Real visual understanding, especially of physics and common sense, is essential for AI systems that interact with the physical world. Current evaluations mostly use real-world videos similar to training data, so high benchmark scores may not reflect real reasoning ability. To address this, we propose negative-control tests using videos that depict physically impossible or logically inconsistent events. We introduce VideoHallu, a synthetic dataset of physics- and commonsense-violating scenes generated with Veo2, Sora, and Kling. It includes expert-annotated question-answer pairs across four categories of violations. Tests of leading VLMs (Qwen-2.5-VL, Video-R1, VideoChat-R1) show that, despite strong results on benchmarks such as MVBench and MMVU, they often miss these violations, exposing gaps in visual reasoning. Reinforcement learning fine-tuning on VideoHallu improves recognition of such violations without reducing standard benchmark performance. Our data is available at https://github.com/zli12321/VideoHallu.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。