无需调参即可在不牺牲任务性能前提下大幅降低机器人能耗
Non-conflicting Energy Minimization in Reinforcement Learning based Robot Control
- 通过双目标梯度投影实现能耗最小化，避免与任务目标冲突
- 在标准基准上实现64%能耗降低，任务表现基本不变
- 方法通用性强，适合各类策略梯度算法，易集成到现有系统
高效机器人控制常需在任务表现与能耗之间取得平衡。强化学习中常见做法是将能耗惩罚直接纳入奖励函数，但需精细调节权重以避免能耗优化损害任务成功率。本文提出一种无超参数的梯度优化方法，可在不干扰任务目标的前提下最小化能耗。受多任务学习启发，该方法在任务与能耗目标间进行策略梯度投影，生成既降低能耗又不影响任务表现的策略更新。我们在DM-Control和HumanoidBench标准运动基准上验证了该方法，实现了64%的能耗减少，同时保持相近的任务性能。此外，我们在Unitree GO2四足机器人上进行了模拟到现实（Sim2Real）迁移实验，验证了节能策略的有效性。该方法可轻松嵌入标准强化学习流程，仅需少量代码修改，适用于任意策略梯度方法，为节能控制提供了比奖励设计更合理的替代方案。
原文摘要 · Abstract (English)
Efficient robot control often requires balancing task performance with energy expenditure. A common approach in reinforcement learning (RL) is to penalize energy use directly as part of the reward function. This requires carefully tuning weight terms to avoid undesirable trade-offs where energy minimization harms task success. In this work, we propose a hyperparameter-free gradient optimization method to minimize energy expenditure without conflicting with task performance. Inspired by recent works in multitask learning, our method applies policy gradient projection between task and energy objectives to derive policy updates that minimize energy expenditure in ways that do not impact task performance. We evaluate this technique on standard locomotion benchmarks of DM-Control and HumanoidBench and demonstrate a reduction of 64% energy usage while maintaining comparable task performance. Further, we conduct experiments on a Unitree GO2 quadruped showcasing Sim2Real transfer of energy efficient policies. Our method is easy to implement in standard RL pipelines with minimal code changes, is applicable to any policy gradient method, and offers a principled alternative to reward shaping for energy efficient control policies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。