用2048游戏测试强化学习在延迟奖励下的表现，新方法让模型最高得分超4万。
2048: Reinforcement Learning in a Delayed Reward Environment
- 设计多步分布强化学习框架，直接优化长期收益
- 新模型H-DQN最高得分为41,828，达成4096方块
- 适合研究稀疏奖励场景的RL算法与游戏智能体开发者
延迟且稀疏的奖励是强化学习代理面临的核心挑战，因其难以将动作的长期效益归因。滑动拼图游戏2048正是此类问题的典型：频繁的小分变化提供即时反馈，却常诱导代理陷入局部最优而全局次优。本文提出统一的分布式多步强化学习框架，旨在直接优化长时序性能。我们基于开源Gym-2048环境对比四种代理：标准DQN、PPO、QR-DQN（分位数回归DQN）及新型Horizon-DQN（H-DQN），后者融合分布学习、双网络结构、噪声网络、优先级回放等技术。实验显示效果显著提升：最大回合得分从DQN的3.988K增至PPO的5.756K、QR-DQN的8.66K，最终达H-DQN的18.21K，成功达成2048方块。进一步扩展后，H-DQN最高得分达41.828K，实现4096方块。结果表明，分布式多步目标显著提升稀疏奖励场景下的性能，提示通过模型规划与课程学习可获更大进展。
原文摘要 · Abstract (English)
Delayed and sparse rewards present a fundamental obstacle for reinforcement-learning (RL) agents, which struggle to assign credit for actions whose benefits emerge many steps later. The sliding-tile game 2048 epitomizes this challenge: although frequent small score changes yield immediate feedback, they often mislead agents into locally optimal but globally suboptimal strategies. In this work, we introduce a unified, distributional multi-step RL framework designed to directly optimize long-horizon performance. Using the open source Gym-2048 environment we develop and compare four agent variants: standard DQN, PPO, QR-DQN (Quantile Regression DQN), and a novel Horizon-DQN (H-DQN) that integrates distributional learning, dueling architectures, noisy networks, prioritized replay, and more. Empirical evaluation reveals a clear hierarchy in effectiveness: max episode scores improve from 3.988K (DQN) to 5.756K (PPO), 8.66K (QR-DQN), and 18.21K (H-DQN), with H-DQN reaching the 2048 tile. Upon scaling H-DQN it reaches a max score 41.828K and a 4096 tile. These results demonstrate that distributional, multi-step targets substantially enhance performance in sparse-reward domains, and they suggest promising avenues for further gains through model-based planning and curriculum learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。