让通用视觉语言模型学会看足球，效果提升超三成。
Domain Adaptation of VLM for Soccer Video Understanding
- 用足球数据和大模型生成指令数据，分阶段微调模型
- 在2万段视频上训练后，问答任务性能提升37.5%
- 动作分类准确率从11.8%升至63.5%，适合体育视频研究者
视觉语言模型（VLM）在多模态任务中表现优异，但现有研究多忽略领域适应问题。本文以足球视频理解为切入点，利用大规模足球数据集与大语言模型生成指令跟随数据，采用课程学习方式迭代微调通用域VLM：先教关键足球概念，再进行问答任务。最终在20,000段视频组成的精选数据集上训练的模型，在足球特定任务中表现显著提升——视觉问答任务相对改进37.5%，下游足球动作分类准确率从11.8%提升至63.5%。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) have demonstrated strong performance in multi-modal tasks by effectively aligning visual and textual representations. However, most video understanding VLM research has been domain-agnostic, leaving the understanding of their transfer learning capability to specialized domains under-explored. In this work, we address this by exploring the adaptability of open-source VLMs to specific domains, and focusing on soccer as an initial case study. Our approach uses large-scale soccer datasets and LLM to create instruction-following data, and use them to iteratively fine-tune the general-domain VLM in a curriculum learning fashion (first teaching the model key soccer concepts to then question answering tasks). The final adapted model, trained using a curated dataset of 20k video clips, exhibits significant improvement in soccer-specific tasks compared to the base model, with a 37.5% relative improvement for the visual question-answering task and an accuracy improvement from 11.8% to 63.5% for the downstream soccer action classification task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。