用音视频回放评估生成游戏，实现多智能体自动优化。
Multi-Agent Game Generation and Evaluation via Audio-Visual Recordings
- 构建音视频回放评估系统，通过多模态模型对比内容优劣。
- 多智能体系统生成游戏代码，利用反馈循环提升质量，胜率显著提高。
- 揭示当前模型难以有效利用高质量素材与视觉反馈，凸显人机差异。
尽管人工智能在生成文本、音频、图像和视频方面表现优异，但创建交互式音视频内容（如电子游戏）仍具挑战性。现有大模型可生成JavaScript游戏与动画，但缺乏自动化评估指标，且难以处理需团队协作数月完成的复杂内容（多轮、多智能体），通常依赖艺术家制作的多媒体资产。为此，我们提出AVR-Eval——一种基于音视频回放（AVRs）的多模态内容质量相对评估指标。该指标由全模态模型（处理文本、视频、音频）比较两个内容的音视频回放，并由文本模型判断优劣。实验表明，AVR-Eval能有效识别优质与错误或不匹配的内容。我们构建了AVR-Agent多智能体系统，从多媒体资产库（音频、图像、3D模型）中生成JavaScript代码。编码智能体选取相关资源，生成多个初始代码，利用AVR-Eval筛选最优版本，并通过音视频回放反馈迭代优化。在游戏与动画实验中，使用AVR-Eval的对抗测试显示，AVR-Agent生成内容对单次生成内容的胜率显著更高。然而，模型在利用自定义资产和AVR反馈方面表现不佳，未体现更高胜率。这揭示出关键差距：人类能有效利用高质量资产与音视频反馈，而当前编码模型尚未具备类似能力，凸显人机内容创作范式的根本差异。
原文摘要 · Abstract (English)
While AI excels at generating text, audio, images, and videos, creating interactive audio-visual content such as video games remains challenging. Current LLMs can generate JavaScript games and animations, but lack automated evaluation metrics and struggle with complex content that normally requires teams of humans working for many months (multi-shot, multi-agents) using assets made by artists. To tackle these issues, we built a new metric and a multi-agent system. We propose AVR-Eval, a relative metric for multimedia content quality using Audio-Visual Recordings (AVRs). An omni-modal model (processing text, video, and audio) compares the AVRs of two contents, with a text model reviewing evaluations to determine superiority. We show that AVR-Eval properly identifies good from broken or mismatched content. We built AVR-Agent, a multi-agent system generating JavaScript code from a bank of multimedia assets (audio, images, 3D models). The coding agent selects relevant assets, generates multiple initial codes, uses AVR-Eval to identify the best version, and iteratively improves it through omni-modal agent feedback from the AVR. We run experiments on games and animations with AVR-Eval (win rate of content A against B). We find that content generated by AVR-Agent has a significantly higher win rate against content made through one-shot generation. However, models struggle to leverage custom assets and AVR feedback effectively, showing no higher win rate. This reveals a critical gap: while humans benefit from high-quality assets and audio-visual feedback, current coding models do not seem to utilize these resources as effectively, highlighting fundamental differences between human and machine content creation approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。