用同步分布的误差信号实现强化学习，媲美反向传播效果
Temporal-Difference Learning Using Distributed Error Signals
- 模拟多巴胺同步分布，每层使用时序差分误差更新
- 在MinAtar和深脑控制套件上性能接近基于反向传播的算法
- 为生物学习机制如何实现复杂任务提供计算支持
生物奖励学习中的一个计算难题是伏隔核（NAc）如何进行信用分配。大量研究认为NAc多巴胺编码了时序差分（TD）误差以学习价值预测。然而，多巴胺以区域均一的同步方式分布，无法支持显式的信用分配（如反向传播所用）。目前尚不清楚仅靠分布式误差是否足以使突触协同更新，从而学习复杂的非线性奖励学习任务。为此，我们设计了一种新的深度Q学习算法——人工多巴胺，从计算上证明：同步分布的、每层的时序差分误差可能足以学习出令人惊讶的复杂强化学习任务。我们在MinAtar、DeepMind Control Suite及经典控制任务上对算法进行了实证评估，结果表明其性能通常可与使用反向传播的深度强化学习算法相媲美。
原文摘要 · Abstract (English)
A computational problem in biological reward-based learning is how credit assignment is performed in the nucleus accumbens (NAc). Much research suggests that NAc dopamine encodes temporal-difference (TD) errors for learning value predictions. However, dopamine is synchronously distributed in regionally homogeneous concentrations, which does not support explicit credit assignment (like used by backpropagation). It is unclear whether distributed errors alone are sufficient for synapses to make coordinated updates to learn complex, nonlinear reward-based learning tasks. We design a new deep Q-learning algorithm, Artificial Dopamine, to computationally demonstrate that synchronously distributed, per-layer TD errors may be sufficient to learn surprisingly complex RL tasks. We empirically evaluate our algorithm on MinAtar, the DeepMind Control Suite, and classic control tasks, and show it often achieves comparable performance to deep RL algorithms that use backpropagation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。