用约束强化学习让水下机器人省电,自动达标不需调参。
Average-Power-Budgeted Underwater Vehicle Control via Constrained Reinforcement Learning

- 将能耗设为物理单位的显式约束,替代传统惩罚项。
- 在12组测试中平均省电14%~65%,最高达64.9%。
- 无需调参,适合各类水下车辆与任务,性能稳定。
水下车辆受限于固定的机载能量预算,推进系统会快速耗尽能源,因此能以更低推进功率完成任务的控制器可直接延长任务范围与续航时间。强化学习虽能生成无需模型的驻位与轨迹跟踪控制器,但仅优化任务精度会导致策略产生振荡且耗能的执行动作。现有方法通过在奖励中减去能量惩罚来缓解,但该方式依赖单一无量纲权重调节任务-能耗平衡:无法设定具体功率目标,需为每辆车和每项任务重新调参,且权重不当反而可能增加能耗。本文提出将节能控制建模为带平均功耗约束的马尔可夫决策过程,采用PPO-Lagrangian算法求解。功率上限以物理单位明确设定,仅需在线更新一个对偶变量即可满足各车辆与任务的约束,无需手动调参。在MarineGym仿真器中的三类车辆、四类任务共十二组实验中,所提能耗约束策略在所有场景下均实现最低功耗,相比仅优化任务的基线降低14%~65%(最高64.9%),且在所有场景下均优于能量奖励基线;在十种设置中保持最平滑控制,仅在一种刻意限制功率的场景下略有精度下降。将能耗作为显式约束,提供了无需调参的节能控制路径,避免了针对每辆车、每项任务的权重搜索。
原文摘要 · Abstract (English)
Underwater vehicles operate from a fixed onboard energy budget that propulsion rapidly depletes, so a controller that completes its task while drawing less thruster power directly extends mission range and endurance. Reinforcement learning yields capable model-free controllers for station-keeping and trajectory tracking, but optimizing task accuracy alone drives the policy toward oscillatory, energy-wasting actuation. The established remedy subtracts an energy penalty from the reward, yet this sets the task-power trade-off through a single weight with no physical units: a target power level cannot be specified, the weight must be re-tuned for every vehicle and task, and a mismatched weight can even raise power. This paper instead formulates energy-efficient underwater control as a constrained Markov decision process in which average thruster power is subject to an explicit budget, solved with a PPO-Lagrangian algorithm. The power level is set by declaring a budget in physical units, and a single dual variable is updated online to meet it for each vehicle and task, without manual weight search. Across three vehicles and four tasks in the MarineGym simulator, the energy-constrained policy draws the least power in all twelve settings, reducing it by 14--65\% (up to 64.9\%) over a task-only baseline and below an energy-reward baseline everywhere, while remaining the smoothest in ten settings and preserving task accuracy except in one deliberately power-limited regime. Imposing energy as an explicit constraint thus offers a tuning-free route to energy-efficient underwater control that needs no per-vehicle, per-task weight search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。