让AI看懂镜头运动的细节变化,支持多动作并行识别。
Temporally Grounded Compositional Camera Motion Understanding via Geometric Knowledge Distillation

- 用几何知识蒸馏技术,在训练中注入镜头运动的几何特征。
- 在4229个真实镜头上实现细粒度、复合型运动识别,近半段含多重运动。
- 模型推理无需运行复杂3D模型,效率提升且准确率不降。
理解镜头运动是视频感知的基础,应用于空间智能和可控视频生成。多模态大模型(MLLM)为此提供了自然接口,但现有方法通常对整段视频打一个或多个标签,忽略了真实镜头运动的两个关键特性:运动可在单镜头内变化,且多种运动可同时发生。因此,我们提出将镜头运动理解为时序定位的组合识别任务,要求模型准确定位运动一致的时间段,并识别每个时间段内活跃的所有运动。我们构建了CamChoreo基准数据集,包含4,229个真实单镜头片段,由专家标注时间区间,使用20个方向感知标签,近一半片段包含复合运动(多重运动并行)。当前MLLM视觉编码器侧重语义内容,缺乏几何证据支持运动判断。直接注入冻结3D基础模型的特征虽有效,但需对每输入运行昂贵的几何模型,称为CamInject。我们提出CamDistill,通过知识蒸馏在训练中将几何知识融入轻量级相机标记,推理时移除3D模型。结果表明,CamDistill在无需运行3D教师模型的情况下达到与直接注入相当的准确率。结合CamChoreo与CamDistill,推动镜头运动理解从片段级标注迈向时序定位、组合识别的新阶段。
原文摘要 · Abstract (English)
Understanding camera motion is fundamental to video perception, with applications in spatial intelligence and controllable video generation. Multimodal large language models (MLLMs) provide a natural interface for this task, but existing work typically assigns one or more labels to an entire clip. Such clip-level recognition overlooks two defining properties of real camera motion: it can change within a shot, and multiple movements can occur simultaneously. We therefore formulate camera-motion understanding as temporally grounded, compositional recognition, which requires a model to localize motion-consistent intervals and identify every movement active within each interval. We introduce CamChoreo, a benchmark of 4,229 real single-shot clips with expert-annotated temporal segments. Its annotations use a compact vocabulary of 20 direction-aware labels, and nearly half of the segments contain compound camera motion, with multiple movement primitives active simultaneously. Recognizing such fine-grained, compositional motion is hard for current MLLMs, whose visual encoders emphasize semantic content rather than the geometric evidence on which camera motion depends. Directly injecting features from a frozen 3D foundation model addresses this gap, but requires running the expensive geometry model on every input; we refer to this baseline as CamInject. We instead propose CamDistill, which distills the same geometric knowledge into lightweight camera tokens during training and removes the 3D model at inference. CamDistill matches the accuracy of direct feature injection without running the 3D teacher at inference. Together, CamChoreo and CamDistill advance camera-motion understanding from clip-level labeling to temporally grounded, compositional recognition. Project page: https://ddz16.github.io/cammotion.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。