让强化学习自动调整时间尺度，提升环境变化时的适应能力
Adaptive Multi-Horizon Reinforcement Learning
- 动态选择多个时间跨度，而非固定折扣因子
- 在三任务连续环境中无需调参即可稳定表现
- 适合需要持续学习与任务切换的场景
复杂多变环境中有效决策需权衡短期与长期后果。传统强化学习采用固定折扣因子，仅支持单一指数衰减时间跨度。但生物体表现出灵活的时间折扣特性，表明有效规划应具备多时间尺度。本文提出一种自适应多时间跨度方法，能动态选择并组合不同时间尺度，实现对奖励结构变化的鲁棒适应，无需手动调节折扣因子。实验表明，在包含三个连续任务的MiniGrid环境中，该方法能自动识别有效折扣因子。结果表明，自适应时间折扣可提升参数效率，并增强人工与类生物学习系统的适应性。
原文摘要 · Abstract (English)
Effective decision-making in complex and changing environments requires balancing short-term and long-term consequences. In reinforcement learning (RL), this trade-off is typically controlled through a fixed discount factor, which imposes a single exponentially discounted temporal horizon. However, biological agents exhibit flexible and adaptive temporal discounting, suggesting that effective planning requires multiple timescales. Here, we propose a multi-horizon approach that adaptively selects and combines temporal horizons, enabling robust adaptation to changes in reward structure without manual discount-factor tuning. This flexibility makes the method particularly suitable for continual learning scenarios involving task switches and varying environmental configurations. Empirically, we demonstrate that our approach identifies effective discount factors across a range of MiniGrid environments, including continual settings composed of three sequentially changing tasks. These results suggest that adaptive temporal discounting can improve parameter efficiency and enhance adaptability in both artificial and biologically inspired learning systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。