arXiv:2601.04778cs.CVcs.AI2026-01被引 6

通过生成反事实视频,解决视频语言模型的动作和时序幻觉问题。

CounterVid: Counterfactual Video Generation for Mitigating Action and Temporal Hallucinations in Video-Language Models

  • 用多模态大模型生成仅在动作或时序上不同的反事实视频。
  • 构建了2.6万对反事实视频数据集,提升动作识别准确率。
  • 适合研究视频理解、幻觉抑制及训练鲁棒多模态模型的研究者。

视频语言模型虽具备强大多模态理解能力,但在动作与时间顺序推理时仍易产生幻觉。现有缓解策略如文本过滤或随机视频扰动,难以根治其依赖语言先验而非细粒度视觉动态的本质问题。本文提出可扩展的反事实视频生成框架,合成仅在动作或时序结构上不同但场景保持一致的视频。该流程结合多模态大模型进行动作提议与编辑指导,利用基于扩散的图像与视频模型大规模生成语义硬负样本。基于此,构建了包含约2.6万对偏好样本的CounterVid数据集,聚焦于动作识别与受控动作序列排序。进一步提出MixDPO,一种联合利用文本与视觉偏好的一体化直接偏好优化方法。在Qwen2.5-VL与InternVL3模型上,MixDPO显著提升动作识别与时序排序性能,在多数标准视频幻觉基准上取得增益,同时基本保持通用视频理解能力。源代码、训练模型与数据集已公开于https://aimagelab.github.io/CounterVid。

原文摘要 · Abstract (English)

Video-language models (VLMs) achieve strong multimodal understanding but remain prone to hallucinations, especially when reasoning about actions and temporal order. Existing mitigation strategies, such as textual filtering or random video perturbations, often fail to address the root cause: over-reliance on language priors rather than fine-grained visual dynamics. We propose a scalable framework for counterfactual video generation that synthesizes videos differing only in actions or temporal structure while preserving scene context. Our pipeline combines multimodal LLMs for action proposal and editing guidance with diffusion-based image and video models to generate semantic hard negatives at scale. Using this framework, we build CounterVid, a synthetic dataset of ~26k preference pairs constructed from short counterfactual action clips and targeting both action recognition and controlled action-sequence ordering. We further introduce MixDPO, a unified Direct Preference Optimization approach that jointly leverages textual and visual preferences. Across Qwen2.5-VL and InternVL3 backbones, MixDPO substantially improves action recognition and temporal ordering, yields gains on most standard video hallucination benchmarks, and largely preserves general video understanding. Our source code, trained models, and dataset are available at https://aimagelab.github.io/CounterVid.

视频生成幻觉抑制多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。