评测视频模型对周期性状态变化的推理能力,发现现有模型普遍缺乏时间理解。
CycliST: A Video Language Model Benchmark for Reasoning on Cyclical State Transitions
- 构建周期性运动与视觉属性变化的合成视频数据集
- 多任务测试显示模型无法稳定识别周期模式和量化信息
- 揭示大模型性能不随规模或架构提升,适合视觉推理研究者
我们提出CycliST,一个新型基准数据集,用于评估视频语言模型(VLM)在周期性状态转换上的文本推理能力。该数据集通过生成具有物体运动和视觉属性周期性变化的合成视频序列,捕捉真实世界过程的核心特征。CycliST采用分层评估体系,通过增加循环物体数量、场景杂乱度和光照条件变化逐步提升难度,挑战模型的时空认知能力。我们在当前最先进的开源与专有VLM上进行广泛实验,发现它们在处理线性与轨道运动等周期动态,以及颜色、尺度等随时间变化的视觉属性时存在显著局限。结果表明,现有VLM难以可靠检测和利用周期模式,缺乏时间理解能力,也无法从场景中提取如运动物体数量等定量信息,暴露出亟待解决的技术差距。具体而言,无一模型在所有任务中持续领先:模型规模或架构与性能无强相关性,且无模型在各任务中表现均衡。通过提供针对性挑战和全面评估框架,CycliST为超越现有水平的周期性模式理解模型铺平了道路。
原文摘要 · Abstract (English)
We present CycliST, a novel benchmark dataset designed to evaluate Video Language Models (VLM) on their ability for textual reasoning over cyclical state transitions. CycliST captures fundamental aspects of real-world processes by generating synthetic, richly structured video sequences featuring periodic patterns in object motion and visual attributes. CycliST employs a tiered evaluation system that progressively increases difficulty through variations in the number of cyclic objects, scene clutter, and lighting conditions, challenging state-of-the-art models on their spatio-temporal cognition. We conduct extensive experiments with current state-of-the-art VLMs, both open-source and proprietary, and reveal their limitations in generalizing to cyclical dynamics such as linear and orbital motion, as well as time-dependent changes in visual attributes like color and scale. Our results demonstrate that present-day VLMs struggle to reliably detect and exploit cyclic patterns, lack a notion of temporal understanding, and are unable to extract quantitative insights from scenes, such as the number of objects in motion, highlighting a significant technical gap that needs to be addressed. More specifically, we find no single model consistently leads in performance: neither size nor architecture correlates strongly with outcomes, and no model succeeds equally well across all tasks. By providing a targeted challenge and a comprehensive evaluation framework, CycliST paves the way for visual reasoning models that surpass the state-of-the-art in understanding periodic patterns.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。