用强化学习让微型机器人在有限算力下自主行走,提升实时性与稳定性。
Control of Microrobots with Reinforcement Learning under On-Device Compute Constraints
- 在微型芯片上训练轻量级神经网络,实现低延迟控制。
- 通过量化和功耗建模,使控制频率提升至每秒100次以上。
- 提出按功耗选步态的新策略,适合资源受限的微机器人系统。
自主微机器人的一项关键能力是能够在复杂地形上稳定移动。本文探索了边缘机器学习方法用于微机器人运动控制,在计算、内存和功耗受限条件下实现设备端低延迟控制。研究基于一个亚厘米级四足微机器人,采用强化学习(RL)训练一个紧凑的FP32多层感知机(MLP)策略,网络结构为两层隐藏层([128, 64]),在大规模并行GPU仿真环境中进行训练,并通过域随机化增强对仿真参数扰动的鲁棒性。随后,研究使用整数(Int8)量化(按张量和按特征)以提高在资源受限硬件上的推理更新速率,并建立基于周期/更新的模型,将硬件功耗预算与可实现的更新频率关联。提出一种资源感知的步态调度视角:在给定设备功耗预算下,选择能最大化期望强化学习奖励且对应可行更新频率的步态模式(快走/中间/疾跑)。最终,将该MLP策略部署于真实世界大型机器人在不平整地形上运行,定性观察到域随机化训练可提升分布外稳定性。本工作未声称实现真实世界大型机器人零样本迁移的实证效果。
原文摘要 · Abstract (English)
An important function of autonomous microrobots is the ability to perform robust movement over terrain. This paper explores an edge ML approach to microrobot locomotion, allowing for on-device, lower latency control under compute, memory, and power constraints. This paper explores the locomotion of a sub-centimeter quadrupedal microrobot via reinforcement learning (RL) and deploys the resulting controller on an ultra-small system-on-chip (SoC), SC$μ$M-3C, featuring an ARM Cortex-M0 microcontroller running at 5 MHz. We train a compact FP32 multilayer perceptron (MLP) policy with two hidden layers ($[128, 64]$) in a massively parallel GPU simulation and enhance robustness by utilizing domain randomization over simulation parameters. We then study integer (Int8) quantization (per-tensor and per-feature) to allow for higher inference update rates on our resource-limited hardware, and we connect hardware power budgets to achievable update frequency via a cycles-per-update model for inference on our Cortex-M0. We propose a resource-aware gait scheduling viewpoint: given a device power budget, we can select the gait mode (trot/intermediate/gallop) that maximizes expected RL reward at a corresponding feasible update frequency. Finally, we deploy our MLP policy on a real-world large-scale robot on uneven terrain, qualitatively noting that domain-randomized training can improve out-of-distribution stability. We do not claim real-world large-robot empirical zero-shot transfer in this work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。