用视觉+语音数据让AI看懂足球比赛，还能解释裁判判罚。
SoccerChat: Integrating Multimodal Data for Enhanced Soccer Game Understanding
- 融合视频与语音数据，构建对话式足球理解框架。
- 在赛事分类和裁判判罚任务上达到可比现有模型的准确率。
- 适合想做智能体育分析或可解释性研究的人使用。
人工智能在体育分析中的应用已推动足球视频理解的革新，实现对复杂比赛动态的实时、自动化洞察。传统方法依赖孤立的数据流，难以捕捉比赛全貌。为此，我们提出SoccerChat——一种整合视觉与文本数据的多模态对话式AI框架，用于增强足球视频理解。基于包含球衣颜色标注和自动语音识别（ASR）转录的SoccerNet数据集，SoccerChat在结构化视频指令数据集上进行微调，以提升比赛理解、事件分类与裁判决策能力。我们在动作分类与裁判决策任务上对SoccerChat进行基准测试，结果显示其在通用足球事件理解方面表现良好，同时在裁判判罚任务中保持了具有竞争力的准确率。研究结果凸显多模态融合在推进足球分析中的重要性，为更交互、可解释的AI体育分析铺平道路。
原文摘要 · Abstract (English)
The integration of artificial intelligence in sports analytics has transformed soccer video understanding, enabling real-time, automated insights into complex game dynamics. Traditional approaches rely on isolated data streams, limiting their effectiveness in capturing the full context of a match. To address this, we introduce SoccerChat, a multimodal conversational AI framework that integrates visual and textual data for enhanced soccer video comprehension. Leveraging the extensive SoccerNet dataset, enriched with jersey color annotations and automatic speech recognition (ASR) transcripts, SoccerChat is fine-tuned on a structured video instruction dataset to facilitate accurate game understanding, event classification, and referee decision making. We benchmark SoccerChat on action classification and referee decision-making tasks, demonstrating its performance in general soccer event comprehension while maintaining competitive accuracy in referee decision making. Our findings highlight the importance of multimodal integration in advancing soccer analytics, paving the way for more interactive and explainable AI-driven sports analysis. https://github.com/simula/SoccerChat
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。