用课程学习训练网球策略,胜率高达98%以上。
Learning Tennis Strategy Through Curriculum-Based Dueling Double Deep Q-Networks
- 分阶段提升对手难度,稳定训练智能体
- 胜率达98%-100%,发球/回球效率超60%
- 适合研究体育策略优化与奖励设计
网球策略优化是涉及层级计分、随机结果、长时程信用分配、身体疲劳和对手适应性的复杂序列决策问题。本文提出一种强化学习框架,结合自定义网球仿真环境与基于课程学习的双网络深度双Q网络(DDQN)。环境模拟完整的网球计分体系(局、盘、分),在回合层面提供十类战术动作选择,包含对称疲劳机制和连续对手技能参数。双重架构将动作价值分解为状态值与优势值,双Q学习减少高估偏差,提升长周期随机环境下的训练稳定性。课程学习将对手难度从0.40逐步提升至0.50,避免固定对手导致的训练崩溃。大量评估显示,训练智能体在平衡对手中胜率98%~100%,对更强对手也表现稳健。发球效率63.0%~67.5%,回球效率52.8%~57.1%。消融实验表明,双重架构与课程学习均对稳定收敛必要,标准DQN基线无法学习有效策略。然而战术分析揭示明显防守倾向,策略偏好避免失误与延长回合,而非主动进攻。这凸显仅以胜率驱动优化在简化体育仿真中的局限性,强调奖励设计对真实体育强化学习的重要性。
原文摘要 · Abstract (English)
Tennis strategy optimization is a challenging sequential decision-making problem involving hierarchical scoring, stochastic outcomes, long-horizon credit assignment, physical fatigue, and adaptation to opponent skill. I present a reinforcement learning framework that integrates a custom tennis simulation environment with a Dueling Double Deep Q-Network(DDQN) trained using curriculum learning. The environment models complete tennis scoring at the level of points, games, and sets, rally-level tactical decisions across ten discrete action categories, symmetric fatigue dynamics, and a continuous opponent skill parameter. The dueling architecture decomposes action-value estimation into state-value and advantage components, while double Q-learning reduces overestimation bias and improves training stability in this long-horizon stochastic domain. Curriculum learning progressively increases opponent difficulty from 0.40 to 0.50, enabling robust skill acquisition without the training collapse observed under fixed opponents. Across extensive evaluations, the trained agent achieves win rates between 98 and 100 percent against balanced opponents and maintains strong performance against more challenging opponents. Serve efficiency ranges from 63.0 to 67.5 percent, and return efficiency ranges from 52.8 to 57.1 percent. Ablation studies demonstrate that both the dueling architecture and curriculum learning are necessary for stable convergence, while a standard DQN baseline fails to learn effective policies. Despite strong performance, tactical analysis reveals a pronounced defensive bias, with the learned policy prioritizing error avoidance and prolonged rallies over aggressive point construction. These results highlight a limitation of win-rate driven optimization in simplified sports simulations and emphasize the importance of reward design for realistic sports reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。