视频大模型常分不清物体移动方向，新方法显著提升准确率。
Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs
- 通过分析信号传递路径，发现方向绑定环节存在漏洞。
- 新方法在合成数据上将准确率从25.9%提升至85.4%。
- 无需真实世界训练数据，即可大幅提升实际场景表现。
视频大语言模型（Video-LLMs）在时间视频理解上进展迅速，但在判断物体在图像平面上的移动方向（左、右、上、下）这一基本感知任务上表现不佳。在简单单对象运动视频中，多数模型准确率接近随机水平，高于随机的结果多由预测偏差导致而非真实方向理解。作者称此为“方向运动盲”（directional motion blindness）。通过追踪信号路径，发现方向信息在视觉编码器、投影器和语言模型隐藏状态中仍线性可解，但读出阶段无法正确绑定到对应答案选项，暴露出“方向绑定缺口”。尽管合成方向指令微调可缓解源域问题，但复杂视觉场景会削弱信号强度，限制跨域泛化能力。为此提出MoDirect数据集族与DeltaDirect诊断驱动目标，直接从相邻帧特征差分预测归一化二维运动向量。在MoDirect-SynBench上，使用DeltaDirect微调使准确率从25.9%提升至85.4%；在真实场景基准MoDirect-RealBench上，相比基线模型提升21.9个百分点，且无需真实世界微调数据，同时保持原有视频理解性能。代码已开源。
原文摘要 · Abstract (English)
Video Large Language Models (Video-LLMs) have made rapid progress on temporal video understanding, yet many fail at a basic perceptual primitive: signed image-plane motion direction. On simple videos of a single object moving left, right, up, or down, most Video-LLMs perform near chance, with above-chance cases largely attributable to prediction biases rather than genuine direction understanding. We call this failure directional motion blindness. We localize the failure by tracing motion direction information through the Video-LLM pipeline. Motion direction remains linearly accessible from the vision encoder, projector, and LLM hidden states, but the readout fails to bind this signal to the correct verbal answer option, revealing a direction binding gap. Although synthetic motion direction instruction tuning reduces this gap on the source domain, motion direction concept vector analysis shows that visual complexity weakens the signal magnitude and limits out-of-domain generalization. We introduce MoDirect, a dataset family for motion direction instruction tuning and evaluation, and DeltaDirect, a diagnosis-driven, projector-level objective that predicts normalized 2-D motion vectors from adjacent-frame feature deltas. On MoDirect-SynBench, instruction tuning with DeltaDirect improves motion direction accuracy from 25.9% to 85.4%. On MoDirect-RealBench, DeltaDirect improves real-world motion direction accuracy by 21.9 points over the vanilla baseline without real-world tuning data, while preserving standard video-understanding performance. Code: https://github.com/KHU-VLL/DeltaDirect
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。