为仿人机器人设计高效安全的强化学习算法,提升训练样本效率与动作稳定性。
Symphony: A Heuristic Normalized Calibrated Advantage Actor and Critic Algorithm in application for Humanoid Robots
- 引入渐变记忆缓冲区和时序优势机制,实现单次更新同时优化策略与价值网络。
- 通过弱化噪声约束动作强度,使机器人在低噪声中安全提升探索熵,避免机械损伤。
- 适用于需高安全性与样本效率的仿人机器人从零训练,尤其适合硬件受限场景。
本文指出,人类学习并非快速完成,而机器人从零训练却常因缺乏耐心导致过度试错。为解决此问题,提出一种名为Symphony的过渡策略确定性演员-评论家算法,兼顾样本效率、样本邻近性与动作安全性。通过“襁褓”正则化机制,以特定方式抑制动作强度但不直接修改动作本身。相比随机算法,采用有限参数噪声,降低动作强度,安全提升熵;仅当动作需求极端时才突破弱噪声。该方法显著降低对环境及机器人机构的损害。引入衰减重放缓冲区,结合双曲正切函数调节采样概率,同时保留近期与长期记忆轨迹。利用时序优势,可在单次传递中更新演员与评论家,并将其整合为单一对象,损失函数可一行实现。
原文摘要 · Abstract (English)
In our work we implicitly suggest that it is a misconception to think that humans learn fast. The learning process takes time. Babies start learning to move in the restricted fluid environment of the womb. Children are often limited by underdeveloped body. Even adults are not allowed to participate in complex competitions right away. However, with robots, when learning from scratch, we often don't have the privilege of waiting for tens of millions of steps. "Swaddling" regularization is responsible for restraining an agent in rapid but unstable development penalizing action strength in a specific way not affecting actions directly. The Symphony, Transitional-policy Deterministic Actor and Critic algorithm, is a concise combination of different ideas for possibility of training humanoid robots from scratch with Sample Efficiency, Sample Proximity and Safety of Actions in mind. It is well known that continuous increase in Gaussian noise without appropriate smoothing is harmful for motors and gearboxes. Compared to Stochastic algorithms, we set limited parametric noise and promote a reduced strength of actions, safely increasing entropy, since the actions are submerged in weaker noise. When actions require more extreme values, actions rise above the weak noise. Training becomes empirically much safer for both the environment around and the robot's mechanisms. We use Fading Replay Buffer: using a fixed formula containing the hyperbolic tangent, we adjust the batch sampling probability: the memory contains a recent memory and a long-term memory trail. Fading Replay Buffer allows us to use Temporal Advantage when we improve the current Critic Network prediction compared to the exponential moving average. Temporal Advantage allows us to update the Actor and Critic in one pass, as well as combine the Actor and Critic in one Object and implement their Losses in one line.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。