用AI自动剪辑游戏精彩片段,无需开发配合,跨游戏通用。
Gameplay Highlights Generation
- 用多模态模型识别游戏中的精彩事件,不依赖游戏开发方。
- 在未见过的游戏画面中检测准确率超90%,低资源游戏也表现良好。
- 适合想快速分享游戏高光时刻的玩家和内容创作者。
本文旨在通过自动从游戏录像中生成吸引眼球的精彩片段,帮助玩家在社交平台高效分享游戏体验。我们采用两阶段方法:先定位视频中有趣事件发生的区间,再进行拼接。构建了自有的游戏事件标注数据集,使用VIA视频标注工具由人工标注关键事件。传统方法如游戏引擎集成或OCR需针对每款游戏定制,成本高且难以跨游戏泛化。我们利用自建数据集对X-CLIP等多模态通用视频理解模型进行微调,结合提示工程提升分类性能。评估显示,该模型在未见过的第一人称射击类游戏片段中检测准确率超过90%。此外,将高资源与低资源游戏联合训练时,低资源游戏表现显著提升,体现迁移学习潜力。为支持部署,采用ONNX库实现跨平台推理,并利用其后处理量化工具降低模型体积与推理时间。基于DirectML后端的ONNX运行时可在Windows系统上实现高效推理。结果表明,自然语言监督可使视频识别模型具备数据高效与高性能特性。
原文摘要 · Abstract (English)
In this work, we enable gamers to share their gaming experience on social media by automatically generating eye-catching highlight reels from their gameplay session Our automation will save time for gamers while increasing audience engagement. We approach the highlight generation problem by first identifying intervals in the video where interesting events occur and then concatenate them. We developed an in-house gameplay event detection dataset containing interesting events annotated by humans using VIA video annotator. Traditional techniques for highlight detection such as game engine integration requires expensive collaboration with game developers. OCR techniques which detect patches of specific images or texts require expensive per game engineering and may not generalize across game UI and different language. We finetuned a multimodal general purpose video understanding model such as X-CLIP using our dataset which generalizes across multiple games in a genre without per game engineering. Prompt engineering was performed to improve the classification performance of this multimodal model. Our evaluation showed that such a finetuned model can detect interesting events in first person shooting games from unseen gameplay footage with more than 90% accuracy. Moreover, our model performed significantly better on low resource games (small dataset) when trained along with high resource games, showing signs of transfer learning. To make the model production ready, we used ONNX libraries to enable cross platform inference. These libraries also provide post training quantization tools to reduce model size and inference time for deployment. ONNX runtime libraries with DirectML backend were used to perform efficient inference on Windows OS. We show that natural language supervision in the X-CLIP model leads to data efficient and highly performant video recognition models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。