价值型强化学习可预测地扩展，突破了其难以规模化的固有认知。
Value-Based Deep RL Scales Predictably
- 通过更新数据比（UTD）控制资源需求的帕累托前沿
- 在给定预算下可预测最优数据与计算分配方案
- 首次实现对强化学习过拟合与塑性损失的有效管理
现代机器学习的成功依赖于数据与算力的扩展，但扩展需具备可预测性：我们希望方法不仅在更多算力或数据下表现更好，且性能可从少量实验中预判，无需进行大规模实验。本文表明，尽管社区普遍认为基于价值的离策略强化学习存在病态行为，但其实际具有可预测的扩展特性。首先，达到特定性能水平所需的数据与算力要求位于由更新-数据比（UTD）控制的帕累托前沿上；通过估计该前沿，可在给定算力时预测所需数据量，或在给定数据量时预测所需算力。其次，我们确定了在总资源预算下达成目标性能的最优数据与算力分配，并据此设定最大化性能的超参数。第三，这一可预测性源于对超参数间关系的先验估计，从而有效缓解强化学习特有的过拟合与塑性损失问题。我们在SAC、BRO和PQL三个算法上，在DeepMind Control、OpenAI Gym和IsaacGym数据集上验证了该方法在更高数据量、算力、预算或性能水平下的外推能力。
原文摘要 · Abstract (English)
Scaling data and compute is critical to the success of modern ML. However, scaling demands predictability: we want methods to not only perform well with more compute or data, but also have their performance be predictable from small-scale runs, without running the large-scale experiment. In this paper, we show that value-based off-policy RL methods are predictable despite community lore regarding their pathological behavior. First, we show that data and compute requirements to attain a given performance level lie on a Pareto frontier, controlled by the updates-to-data (UTD) ratio. By estimating this frontier, we can predict this data requirement when given more compute, and this compute requirement when given more data. Second, we determine the optimal allocation of a total resource budget across data and compute for a given performance and use it to determine hyperparameters that maximize performance for a given budget. Third, this scaling is enabled by first estimating predictable relationships between hyperparameters, which is used to manage effects of overfitting and plasticity loss unique to RL. We validate our approach using three algorithms: SAC, BRO, and PQL on DeepMind Control, OpenAI gym, and IsaacGym, when extrapolating to higher levels of data, compute, budget, or performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。