用模拟电阻网络实现低功耗强化学习，突破传统数字硬件限制。
Physical Reinforcement Learning
- 将Q-learning适配到模拟的自调节电阻网络中
- 在两个简单任务上成功实现强化学习训练
- 适合能源受限、需抗损的生物启发式智能系统
数字计算机功耗高且对故障敏感，在不确定环境中难以用于能量受限的自主代理。最近发展的对比局部学习网络（CLLNs）——由自调节非线性电阻构成的模拟网络——具有固有的低功耗特性且能容忍物理损伤，但此前仅用于监督学习。本文展示了将适应于模拟CLLN的Q-learning在两个简单强化学习任务上的成功应用。该方法明确了实现各种强化学习工具（如策略函数、价值函数）所需的关键组件（除被训练网络外），其中部分组件（如策略与价值函数）在该系统中更自然，而其他如经验回放缓冲区则较难实现。我们讨论了数字硬件所需的物理安全性，而CLLN可忽略此要求，生物系统亦无法依赖；同时强调了在生物学中重要且可在CLLN中训练的次级目标，这些目标在传统数字计算机中意义不大。
原文摘要 · Abstract (English)
Digital computers are power-hungry and largely intolerant of damaged components, making them potentially difficult tools for energy-limited autonomous agents in uncertain environments. Recently developed Contrastive Local Learning Networks (CLLNs) - analog networks of self-adjusting nonlinear resistors - are inherently low-power and robust to physical damage, but were constructed to perform supervised learning. In this work we demonstrate success on two simple RL problems using Q-learning adapted for simulated CLLNs. Doing so makes explicit the components (beyond the network being trained) required to enact various tools in the RL toolbox, some of which (policy function and value function) are more natural in this system than others (replay buffer). We discuss assumptions such as the physical safety that digital hardware requires, CLLNs can forgo, and biological systems cannot rely on, and highlight secondary goals that are important in biology and trainable in CLLNs, but make little sense in digital computers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。