实测3.3万次实验，发现常用强化学习配置不可靠，新方案更稳定。
Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitioners
- 用真实水处理厂任务测试3.3万次实验，分析组件影响
- 高斯分布+路径梯度配置最不可靠，易受超参影响
- 有界分布+自适应更新更稳健，适合工程落地
强化学习正被越来越多用于控制真实世界系统，如聚变等离子体、自动驾驶、药物发现和饮用水处理，这些场景对可靠性要求极高且调参预算有限。演员-评论家算法共享一系列设计选择，包括策略更新方式、动作分布表示、梯度估计方法以及与价值估计器的更新频率。基于一个来自真实水处理厂的控制任务，我们通过超过33,000次实验，分析这些组件对运行间变异性和超参数敏感性的影响。发现常见的默认设置，如使用路径梯度估计器的高斯动作分布，是可靠性最低的配置之一；而采用有界动作分布并配合自适应更新调度的方案，在多种环境下仍保持鲁棒性。这些结果为科学与工程领域的从业者提供了实证指导，帮助其在将演员-评论家方法迁移到新现实控制场景时做出组件级决策。
原文摘要 · Abstract (English)
Reinforcement learning is increasingly being considered for controlling real-world systems, from fusion plasma and autonomous vehicles to drug discovery and drinking water treatment, where reliability is essential and tuning budgets are limited. Actor-critic algorithms share a set of design decisions, such as how the policy is updated, how it represents the distribution over actions, how its gradient is estimated, and how often it is updated relative to the value estimator. Using a control task derived from a real water treatment plant, we analyze over 33,000 experiments to determine how these components affect variability across runs and sensitivity to hyperparameters. Common defaults, such as Gaussian action distributions with pathwise gradient estimators, are among the least reliable configurations, whereas bounded distributions with adaptive update schedules remain robust across a wide range of settings. These findings offer empirical guidance to practitioners across scientific and engineering domains for understanding and making component-level decisions when adapting actor-critic methods to new real-world control settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。