构建10万条跨模态推理数据，让AI更懂视频中声音与画面的关联。
OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains
- 用结构化脚本统一描述音画内容，确保实体跨片段一致
- 基于线索生成问答对，支持长时序和深度跨模态推理
- 适配多模型微调，显著提升视频理解性能
当前音频-视觉问答（QA）自动化流程普遍采用“视频-字幕-QA”范式，但通常将视频切分为短片段，分别生成音视频描述。这种解耦处理破坏了声音与其视觉来源的内在关联，且独立片段处理常导致同一实体在不同段落描述不一致。此外，将长文本理解与问答合成一步完成，使模型局限于局部事件，难以生成具有长期时间关联和深度跨模态推理的提问。为此，我们提出一个自动化数据引擎，包含两个机制：(1) 实体锚定视频脚本（Entity-Anchored Video Scripting），将视频转化为结构化脚本，包含摘要、主要实体列表及分段音视频描述。实体列表作为全局先验，确保跨片段指代一致性，并重建音视频关联；(2) 线索引导问答生成（Clue-Guided QA Generation），引导模型首先从脚本中挖掘跨片段多模态线索，再基于高价值线索生成问答对。基于此流程，我们构建了指令微调数据集 OmniVideo-100K 与人工验证测试集 OmniVideo-Test。在该数据集上微调 VITA-1.5、Qwen2.5-Omni-7B 与 Qwen3-Omni-30B 模型，在 OmniVideo-Test 上取得最高达 20.59% 的性能提升，展现出强泛化能力（在 Daily-Omni 与 JointAVBench 等基准上最高提升达 12.64%）。
原文摘要 · Abstract (English)
Current automated pipelines for audio-visual Question Answering (QA) generally adopt a ``video-caption-QA'' paradigm. However, these methods typically segment videos into short clips and generate separate descriptions for audio and visual modalities. This decoupled processing severs inherent associations between sounds and their visual sources, while independent clip processing often causes inconsistent descriptions of the same entity across segments. Furthermore, coupling long-text comprehension and QA synthesis into a single step often restricts models to localized events, yielding questions lacking long-term temporal connections and deep cross-modal reasoning. To address these issues, we propose an automated data engine featuring two mechanisms: (1) \textbf{Entity-Anchored Video Scripting} transforms videos into structured scripts, comprising summaries, main entity lists, and segment-wise audio-visual descriptions. The entity list serves as a global prior to ensure cross-segment referential consistency and reconstruct audio-visual associations. (2) \textbf{Clue-Guided QA Generation} prompts models to first mine cross-segment, multimodal clues from the script, and subsequently generate QA pairs based on these high-value clues. Leveraging this pipeline, we construct the instruction-tuning dataset \textbf{OmniVideo-100K} and a human-verified test set, \textbf{OmniVideo-Test}. Fine-tuning VITA-1.5, Qwen2.5-Omni-7B and Qwen3-Omni-30B on OmniVideo-100K yields performance gains of up to 20.59% on OmniVideo-Test, demonstrating strong generalization (up to 12.64% improvements) across established benchmarks like Daily-Omni and JointAVBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。