用视觉语言模型为任意游戏录像生成实时解说,无需特定训练
Content Based Video Narration of Gameplay with Vision Language Models
- 用九宫格图像让模型感知动作,降低数据负载
- 通过上下文提示避免重复描述静态画面,提升连贯性
- 支持本地语音合成,适合实时游戏解说场景
实时游戏解说极为稀缺:仅存在于职业电竞转播中,其他场合几乎不存在。本文提出一种基于内容的视频解说系统,利用通用视觉语言模型(VLM)和文本转语音(TTS)后端,为任意游戏录像生成类电竞风格的语音解说,无需游戏特定工具、引擎数据或任务专用训练。系统依赖三个机制:时序马赛克打包将均匀采样的九帧合并为一个3×3图像,使图像原生的VLM能以单个图像输入推理运动信息;上下文条件提示将最近K条解说作为历史记录重播,抑制静态场景下的重复描述;时长条件生成与弹性对齐在提示中约束解说长度,并通过时间缩放或对称填充音频,确保每段语音精准匹配对应片段,实现帧级精确混音而无需强制对齐器。系统支持云端TTS或苹果芯片上6位量化40亿参数的本地TTS模型,实现全本地语音生成。我们报告了对即时战略游戏画面的定性案例研究,成本模型显示马赛克使每分钟图像负载降低9倍,并坦诚披露了观测到的失败模式:虚构游戏状态、马赛克导致分辨率下降、时间缩放引发语调失真。系统已开源,提供可复现基线,完整版本将报告量化评估协议。
原文摘要 · Abstract (English)
Live game commentary is scarce: it exists for professional esports broadcasts and almost nowhere else. We present a content-based video narration system that produces spoken, esports-style commentary for arbitrary gameplay recordings using a general-purpose vision-language model (VLM) and a text-to-speech back end, with no game-specific instrumentation, no engine telemetry, and no task-specific training. Three mechanisms carry the system. Temporal mosaic packing arranges nine uniformly sampled frames into a single 3x3 image, letting an image-native VLM reason about motion while consuming one image payload per segment instead of nine. Context-conditioned prompting replays the K most recent narrations as assistant-role history, suppressing the repetition that dominates per-segment captioning of static scenes. Duration-conditioned generation and elastic alignment constrain narration length in the prompt, then time-scale or symmetrically pad the synthesized audio so each utterance fills its segment slot exactly, giving frame-accurate muxing without a forced aligner. The implementation supports either cloud TTS or a 6-bit quantized 4B-parameter on-device TTS model on Apple silicon, making the speech stage fully local. We report a qualitative case study on real-time strategy footage, a cost model showing the mosaic reduces per-minute image payloads by 9x, and a candid account of observed failure modes - hallucinated game state, resolution loss from mosaicking, and prosody artifacts from time-scaling. We release the system as a reproducible baseline, with an evaluation protocol for the quantitative study a full version will report.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。