用合成视频测试模型跨视频推理能力,发现当前模型远不如人类。
SYNCR: A Cross-Video Reasoning Benchmark with Synthetic Grounding

- 构建可控合成数据集,程序化验证空间时间物理真实
- 最佳模型跨视频推理准确率仅52.5%,人类达89.5%
- 揭示模型在物理运动追踪等细粒度任务上严重不足
多模态大语言模型在单视频理解上进展迅速,但跨独立视频流的推理能力仍不明确。现有多视频基准大多依赖人工标注的真实视频,难以精确控制时空与物理真实,不利于诊断模型失败原因。我们提出SYNCR,一个基于程序化验证的跨视频推理合成基准。使用Habitat、Kubric和CLEVRER模拟器引擎构建,包含8,163组多视频问答对,覆盖9,650个独特视频。评估涵盖八个任务,分为四个诊断维度:时间对齐、空间追踪、比较推理与整体整合。对主流开源与闭源模型的零样本测试显示,最佳模型平均准确率仅52.5%,远低于人类89.5%的基准。模型在时间排序上表现尚可,但在物理与空间推理上表现差,最优模型在运动比较任务中仅达26.0%准确率。参数量扩大与专用推理训练能提升时间对齐能力,但无法稳定改善细粒度物理追踪或全局空间整合。初步的模拟到真实相关性分析表明,若干SYNCR任务可反映真实多视频基准上的模型趋势,同时暴露现有评估未涵盖的推理能力。代码已公开于https://github.com/SaraGhazanfari/SYNCR。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have made rapid progress in single-video understanding, yet their ability to reason across multiple independent video streams remains poorly understood. Existing multi-video benchmarks rely largely on human-annotated real-world footage, limiting the precision of spatial, temporal, and physical ground truth and making it difficult to diagnose model failures. We introduce SYNCR, a controlled synthetic benchmark for cross-video reasoning with programmatically verified grounding. Built using Habitat, Kubric, and CLEVRER simulator engines, SYNCR contains 8,163 multi-video question-answer pairs grounded in 9,650 unique videos. It evaluates MLLMs across eight tasks spanning four diagnostic pillars: Temporal Alignment, Spatial Tracking, Comparative Reasoning, and Holistic Synthesis. Our zero-shot evaluation of leading open- and closed-weight MLLMs reveals a substantial gap between current models and humans: the best model achieves only 52.5% average accuracy, compared to an 89.5% human baseline. Models perform relatively well on temporal ordering but struggle with precise physical and spatial reasoning, with the best model reaching only 26.0% accuracy on Kinematic Comparison. We further find that parameter scaling and reasoning-specialized post-training improve temporal alignment capabilities, but do not reliably address fine-grained physical tracking or global spatial synthesis. Finally, an exploratory sim-to-real correlation analysis suggests that several SYNCR tasks track model-level trends on real-world multi-video benchmarks, while also exposing reasoning capabilities underrepresented by existing evaluations. Code available at https://github.com/SaraGhazanfari/SYNCR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。