模型搜索未必提升性能，抑制过估计偏差才是关键。
The Surprising Difficulty of Search in Model-Based Reinforcement Learning
- 用集成价值函数取最小值来缓解过估计偏差
- 即使模型精准，搜索仍可能降低性能
- 在多个基准上达到当前最优效果
本文研究模型基础强化学习中的搜索问题。传统观点认为长期预测与误差累积是主要障碍，我们提出不同看法：搜索并非可直接替代学习策略的万能方案。令人意外的是，即便模型高度准确，搜索仍可能损害性能。我们发现，缓解过估计偏差的重要性超过提升模型或价值函数的准确性。基于此，提出通过取多个价值函数的最小值有效解决该偏差，使搜索表现显著提升，在多个主流基准任务中实现当前最优性能。
原文摘要 · Abstract (English)
This paper investigates search in model-based reinforcement learning (RL). Conventional wisdom holds that long-term predictions and compounding errors are the primary obstacles for model-based RL. We challenge this view, showing that search is not a drop-in replacement for a learned policy. Surprisingly, we find that search can harm performance even when the model is highly accurate. Instead, we show that mitigating overestimation bias matters more than improving model or value function accuracy. Building on this insight, we identify that taking the minimum over an ensemble of value functions effectively addresses this bias and enables effective search, achieving state-of-the-art performance across multiple popular benchmark domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。