用Q-集合提升扩散策略稳定性,实现实时游戏部署。
Real-Time Diffusion Policies for Games: Enhancing Consistency Policies with Q-Ensembles
- 结合一致性模型与Q-集合,通过不确定性估计增强价值函数
- 推理速度达60赫兹,是当前最优扩散策略的3倍
- 适合需要快速响应和多模式行为的游戏智能体
扩散模型在捕捉游戏智能体复杂多模态动作分布方面表现优异,但其推理速度慢,难以在实时游戏环境中部署。尽管一致性模型可实现单步生成,但在策略学习中常面临训练不稳定和性能下降问题。本文提出CPQE(一致性策略与Q-集合),通过Q-集合进行不确定性估计,提供更可靠的值函数逼近,显著提升训练稳定性和性能,优于经典双Q网络方法。在多个游戏场景的大量实验表明,CPQE实现高达60赫兹的推理速度,远超当前最优扩散策略的20赫兹,同时保持与多步扩散方法相当的性能。相比现有的一致性模型方法,CPQE在全程学习中表现出更高奖励与更强训练稳定性。结果表明,CPQE为游戏等实时应用中部署基于扩散的策略提供了实用解决方案,兼具多模态行为建模与快速推理能力。
原文摘要 · Abstract (English)
Diffusion models have shown impressive performance in capturing complex and multi-modal action distributions for game agents, but their slow inference speed prevents practical deployment in real-time game environments. While consistency models offer a promising approach for one-step generation, they often suffer from training instability and performance degradation when applied to policy learning. In this paper, we present CPQE (Consistency Policy with Q-Ensembles), which combines consistency models with Q-ensembles to address these challenges.CPQE leverages uncertainty estimation through Q-ensembles to provide more reliable value function approximations, resulting in better training stability and improved performance compared to classic double Q-network methods. Our extensive experiments across multiple game scenarios demonstrate that CPQE achieves inference speeds of up to 60 Hz -- a significant improvement over state-of-the-art diffusion policies that operate at only 20 Hz -- while maintaining comparable performance to multi-step diffusion approaches. CPQE consistently outperforms state-of-the-art consistency model approaches, showing both higher rewards and enhanced training stability throughout the learning process. These results indicate that CPQE offers a practical solution for deploying diffusion-based policies in games and other real-time applications where both multi-modal behavior modeling and rapid inference are critical requirements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。