让视频语言模型更关注动作，提升小样本场景下的表现。
LocoMotion: Learning Motion-Focused Video-Language Representations
- 用合成运动生成带动作描述的图文对，聚焦局部运动变化。
- 在有限数据下微调时，下游任务性能显著优于传统方法。
- 通过动词变体增强多样性，打通基础动作与高级语义的联系。
本文致力于学习以动作为核心的视频-语言表示。现有方法多依赖空间信息，仅识别物体和场景即可匹配标题。我们提出 LocoMotion，通过向视频添加合成运动，并利用运动参数生成对应的动作描述性标题，实现对运动过程的关注。此外，引入动词变体改写策略，提升标题多样性，学习基础动作与高层动词之间的关联。实验表明，该方法在多种下游任务中表现优异，尤其在微调数据有限时优势明显。代码已公开：https://hazeldoughty.github.io/Papers/LocoMotion/
原文摘要 · Abstract (English)
This paper strives for motion-focused video-language representations. Existing methods to learn video-language representations use spatial-focused data, where identifying the objects and scene is often enough to distinguish the relevant caption. We instead propose LocoMotion to learn from motion-focused captions that describe the movement and temporal progression of local object motions. We achieve this by adding synthetic motions to videos and using the parameters of these motions to generate corresponding captions. Furthermore, we propose verb-variation paraphrasing to increase the caption variety and learn the link between primitive motions and high-level verbs. With this, we are able to learn a motion-focused video-language representation. Experiments demonstrate our approach is effective for a variety of downstream tasks, particularly when limited data is available for fine-tuning. Code is available: https://hazeldoughty.github.io/Papers/LocoMotion/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。