用少量视频数据微调图像模型，就能实现顶尖视频字幕效果。
Pretrained Image-Text Models are Secretly Video Captioners
- 仅用6000对视频文本数据，拼接帧后微调图像模型
- 在MSRVTT、MSVD上排名第二，VATEX上第三
- 适合资源有限却需高效视频字幕的场景
开发视频字幕模型成本高昂，视频动态特性也增加了多模态建模难度。但研究发现，通过极低计算开销且无需复杂调整视频动态结构，仅需微调图像模型即可超越多个专用视频字幕系统。该方法通过仅6,000个视频-文本对微调BLIP2，并简单拼接帧，在主流基准测试中表现优异：在MSRVTT和MSVD上位列第二，在VATEX上位列第三。本研究从资源优化角度关注三个核心因素：模型规模优化、数据效率最大化及强化学习融合。结果表明，轻量级图像模型适配策略可媲美当前最优视频字幕系统，为低资源场景提供实用解决方案。
原文摘要 · Abstract (English)
Developing video captioning models is computationally expensive. The dynamic nature of video also complicates the design of multimodal models that can effectively caption these sequences. However, we find that by using minimal computational resources and without complex modifications to address video dynamics, an image-based model can be repurposed to outperform several specialised video captioning systems. Our adapted model demonstrates top tier performance on major benchmarks, ranking 2nd on MSRVTT and MSVD, and 3rd on VATEX. We transform it into a competitive video captioner by post training a typical image captioning model BLIP2 with only 6,000 video text pairs and simply concatenating frames (significantly fewer data than other methods), which use 2.5 to 144 million pairs. From a resource optimization perspective, this video captioning study focuses on three fundamental factors: optimizing model scale, maximizing data efficiency, and incorporating reinforcement learning. This extensive study demonstrates that a lightweight, image based adaptation strategy can rival state-of-the-art video captioning systems, offering a practical solution for low-resource scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。