改进强化学习中的经验回放，让失败搜索也变有效。
Adaptable Hindsight Experience Replay for Search-Based Learning
- 动态调整失败路径的目标和策略目标，提升学习效率
- 在方程发现任务中超越纯监督或强化学习方法
- 适合需要高效探索的复杂搜索问题研究者
类似AlphaZero的蒙特卡洛树搜索系统最初用于双人对弈游戏，通过神经网络指导实现探索与利用的动态平衡。该组合也适用于经典搜索问题。然而，原始训练方式依赖模拟结果，在奖励稀疏场景下受限，尤其在早期阶段网络尚无法提供有效指导时。赫因特经验回放（HER）通过将搜索树中的失败轨迹重标注为监督信号来解决此问题。本文提出可调节的赫因特经验回放（Adaptable HER），将HER与AlphaZero融合，支持灵活调整重标注目标、策略目标及轨迹选择策略。实验包括方程发现任务，表明调节能力带来显著性能提升，优于纯监督学习或强化学习。
原文摘要 · Abstract (English)
AlphaZero-like Monte Carlo Tree Search systems, originally introduced for two-player games, dynamically balance exploration and exploitation using neural network guidance. This combination makes them also suitable for classical search problems. However, the original method of training the network with simulation results is limited in sparse reward settings, especially in the early stages, where the network cannot yet give guidance. Hindsight Experience Replay (HER) addresses this issue by relabeling unsuccessful trajectories from the search tree as supervised learning signals. We introduce Adaptable HER (\ours{}), a flexible framework that integrates HER with AlphaZero, allowing easy adjustments to HER properties such as relabeled goals, policy targets, and trajectory selection. Our experiments, including equation discovery, show that the possibility of modifying HER is beneficial and surpasses the performance of pure supervised or reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。