arXiv:2603.13119cs.CVcs.AI2026-03被引 2

让视频大模型学会理解镜头运动,提升影视级视觉感知能力。

Geometry-Guided Camera Motion Understanding in VideoLLMs

  • 构建合成数据集与VQA评测,显式建模镜头运动
  • 发现现有模型对镜头动作识别错误率高,深层网络特征弱
  • 轻量级几何提取+结构化提示注入,无需微调即可增强

镜头运动是塑造视觉感知与影视风格的基本几何信号,但当前视频多模态模型(VideoLLMs)很少显式建模,且在细粒度运动类型识别上表现不佳。本文提出评估-诊断-注入框架:构建大规模合成数据集CameraMotionDataset,将镜头运动建模为约束感知的多标签识别任务,并建立视觉问答基准CameraMotionVQA。在多个现成VideoLLMs上测试发现其运动识别存在显著错误。对Qwen2.5-VL视觉编码器的探针实验表明,镜头运动线索在深层ViT块中表示薄弱,解释了失败原因。为此,我们提出轻量、模型无关的流水线:从3D基础模型提取几何镜头信息,用时序分类器预测有约束的运动类型,并通过结构化提示注入下游模型推理。实验证明该方法显著提升运动识别准确率和模型对镜头动作的响应能力,验证了几何驱动特征提取与结构化提示的有效性。数据集与基准已公开于https://hf.co/datasets/fengyee/camera-motion-dataset-and-benchmark。

原文摘要 · Abstract (English)

Camera motion is a fundamental geometric signal that shapes visual perception and cinematic style, yet current video-capable vision-language models (VideoLLMs) rarely represent it explicitly and often fail on fine-grained motion primitives. We address this gap with a framework of $\textbf{benchmarking}$, $\textbf{diagnosis}$, and $\textbf{injection}$. We curate $\textbf{CameraMotionDataset}$, a large-scale synthetic dataset with explicit camera control, formulate camera motion as constraint-aware multi-label recognition, and construct a VQA benchmark--$\textbf{CameraMotionVQA}$. Across diverse off-the-shelf VideoLLMs, we observe substantial errors in recognizing camera motion primitives. Probing experiments on a Qwen2.5-VL vision encoder suggest that camera motion cues are weakly represented, especially in deeper ViT blocks, helping explain the observed failure modes. To bridge this gap without costly training or fine-tuning, we propose a lightweight, model-agnostic pipeline that extracts geometric camera cues from 3D foundation models (3DFMs), predicts constrained motion primitives with a temporal classifier, and injects them into downstream VideoLLM inference via structured prompting. Experiments demonstrate improved motion recognition and more camera-aware model responses, highlighting geometry-driven cue extraction and structured prompting as practical steps toward a camera-aware VideoLLM and VLA system. The dataset and benchmark is publicly available at https://hf.co/datasets/fengyee/camera-motion-dataset-and-benchmark.

视频理解镜头运动多模态提示工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。