构建视频运动理解新基准，推动视觉语言模型精准捕捉细微动作
MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models
- 设计六类运动相关问题，覆盖真实场景视频数据
- 现有模型在细粒度运动理解上表现较差，尤其受限于序列长度
- 提出编码器融合新方法，提升模型对运动细节的感知能力
近年来，视觉语言模型（VLMs）在视频理解方面取得显著进展，但对细粒度运动的理解能力仍严重不足。为此，我们提出MotionBench，一个全面评估视频理解模型细粒度运动感知能力的基准测试。该基准通过六类运动导向的问题类型，结合多源数据，覆盖真实世界视频内容。实验表明，现有VLM在细粒度运动理解上表现不佳。为在有限序列长度下增强模型对运动细节的感知，我们系统考察了针对视频特征压缩优化的VLM架构，并提出一种新颖高效的跨编码器（Through-Encoder, TE）融合方法。实验显示，更高帧率输入与TE融合可显著提升运动理解性能，但仍存在巨大改进空间。本基准旨在引导和激励更强大视频理解模型的发展，强调细粒度运动理解的重要性。
原文摘要 · Abstract (English)
In recent years, vision language models (VLMs) have made significant advancements in video understanding. However, a crucial capability - fine-grained motion comprehension - remains under-explored in current benchmarks. To address this gap, we propose MotionBench, a comprehensive evaluation benchmark designed to assess the fine-grained motion comprehension of video understanding models. MotionBench evaluates models' motion-level perception through six primary categories of motion-oriented question types and includes data collected from diverse sources, ensuring a broad representation of real-world video content. Experimental results reveal that existing VLMs perform poorly in understanding fine-grained motions. To enhance VLM's ability to perceive fine-grained motion within a limited sequence length of LLM, we conduct extensive experiments reviewing VLM architectures optimized for video feature compression and propose a novel and efficient Through-Encoder (TE) Fusion method. Experiments show that higher frame rate inputs and TE Fusion yield improvements in motion understanding, yet there is still substantial room for enhancement. Our benchmark aims to guide and motivate the development of more capable video understanding models, emphasizing the importance of fine-grained motion comprehension. Project page: https://motion-bench.github.io .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。