用强化学习动态调控大模型训练能耗，显著降耗提效。
Cutting AI Datacenter Energy with Reinforcement Learning: Measured Power Control of LLM Training from One GPU to the Fleet
- 用PPO元控制器实时调整生成参数以匹配实测功耗
- 72B模型下提升18.1%输出量，能效提升26.2%
- 适合需要节能降本的大型AI训练部署场景
强化学习后训练主导现代语言模型开发，但其在GPU硬件上的功耗特性尚未被系统刻画。数据中心通常采用无工作负载感知的静态功耗限制和被动降频机制，导致硬件性能被无差别压制。我们对7B、14B和72B规模的GRPO训练进行了半秒级功耗采样（共38万+样本），部署于1至4块A100 GPU上，并训练一个PPO元控制器，通过调节工作负载自身的生成参数来适配实际功耗。在7B模型500步完整轨迹中，该控制器使功耗越限减少89.8%，同时提升18.1%的输出量与26.2%的能效（每兆瓦时生成的令牌数）。在72B模型上首次部署时未达预期，经诊断为模型分片导致执行器控制权丧失；通过执行器权威性扫描发现，相同参数在并发生成模式下仍保留17-22%的控电能力，揭示了占用率与负载体量之间的权衡原则。据此重建的控制器在三轮72B线上运行中表现优异：相较静态安全基线，输出量提升35.7%，功耗越限仅2.27±1.08%，比无控制运行减少87.2%的越限事件，且在受约束控制器中实现最优平均吞吐与单位能耗产出。在真实测量窗口下，72B模型瞬态越限从半秒分辨率的23.6%降至30秒时的1.6%，5分钟时归零。16块GPU集群组合下，30秒及以上窗口实现零越限，峰值需求仅为标称功率的50%-56%。对于此类集群配置，约两倍超额订阅看似可行，需操作员验证。本文量化了经济与碳排放影响，并提出低成本运营试点方案。
原文摘要 · Abstract (English)
Reinforcement-learning post-training dominates modern language-model development, yet its power behavior on GPU hardware has not been characterized, and datacenters manage GPU power with workload-blind mechanisms, static caps and reactive throttling, that slow hardware indiscriminately. We instrument GRPO training with half-second power telemetry at 7B, 14B, and 72B scales on one to four A100s (380,000+ samples), and train a PPO meta-controller that adapts the workload's own generation parameters to measured power. Against the full 500-step 7B trace, the controller cuts power-limit violations by 89.8% while increasing token output by 18.1% and energy efficiency by 26.2% (tokens per MWh). Deployed live at 72B, the same controller family yields replicated null results, diagnosed as the group-size actuator losing authority under model sharding. An actuator-authority sweep shows the same parameters applied as generation concurrency retain 17-22% power authority, isolating an occupancy-versus-volume principle; a controller rebuilt on that actuator controls a live 72B rollout-generation workload across three replications: 35.7% more output than a static safe baseline at 2.27 +/- 1.08% budget violations, 87.2% fewer violations than uncontrolled operation, and the best mean throughput and energy per token among constrained controllers, with an adaptive threshold rule matching it in one of three operating conditions. Under realistic measurement windows the original 72B transients fall from 23.6% at half-second resolution to 1.6% at 30 s and zero at 5 min; a composed 16-GPU fleet shows zero violations at 30 s and longer, with peak demand at 50-56% of nameplate. For this fleet mix, roughly twofold oversubscription of nameplate appears feasible, subject to operator validation. We quantify the economic and carbon consequences and specify a low-cost operator pilot.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。