规则基控制器在资源调度中表现优于主流强化学习算法。
When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Control
- 用可复现基准测试对比六种强化学习算法与调优规则基控制器。
- 规则基控制器在全部六类负载下成本最低，仅在突发流量上稍逊。
- 算法性能随负载变化波动大，无最优算法，关键在基线调校与评估设计。
在可复现的RLScale-Bench基准测试中，我们评估了六种主流深度强化学习（DRL）算法（PPO、DQN、A2C、SAC、TD3、DDPG）在自适应资源控制中的表现，对比其与经调优的规则基控制器在六类工作负载和五次随机种子下的成本与服务级别约束满足情况。所有算法均在相同架构、训练预算与奖励函数下进行对比。结果显示：（i）调优后的规则基控制器在所有负载下成本最低，仅在突发与闪流负载上略逊于最强的RL代理；（ii）离散动作算法比连续动作算法在约束违反上低一到两个数量级，源于动作空间不匹配；（iii）无单一算法在所有负载中占优，排名变动最大达四位。研究揭示：强化学习在资源控制中的瓶颈并非算法选择，而是基线校准、奖励工程与真实评估协议的设计。
原文摘要 · Abstract (English)
A properly calibrated rule-based autoscaler can beat every one of six mainstream deep reinforcement learning (DRL) algorithms on cost across every workload we test - so when, if ever, does DRL actually help? We study this in RLScale-Bench, a reproducible benchmark and evaluation protocol for DRL on adaptive resource control, where an agent allocates compute to a dynamic workload under cost and service-level constraints. We evaluate PPO, DQN, A2C, SAC, TD3, and DDPG under matched architectures, training budgets, and reward functions against a calibrated rule-based baseline across six workload patterns and five seeds (240 runs), instantiate the benchmark on Kubernetes Horizontal Pod Autoscaling, and probe distribution-shift generalization. Three findings challenge common assumptions: (i) the calibrated controller achieves the lowest cost on all six workloads, though it trails the best RL agents on bursty and flash traffic; (ii) discrete-action algorithms outperform continuous-action ones by one to two orders of magnitude in constraint violations due to action-space mismatch; and (iii) no single algorithm dominates across workloads, with rankings shifting by up to four positions. The bottleneck in RL-based resource control is not algorithm selection but baseline calibration, reward engineering, and realistic evaluation protocols.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。