提出ARQ方法，在无反向传播情况下实现高效强化学习。
Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functions
- 用动作条件化的均方根值函数替代传统Q函数
- 在MinAtar和DeepMind Control Suite上超越现有无反向传播方法
- 兼具生物合理性与高性能，适合追求高效训练的RL研究者
最近提出的前向-前向（FF）算法采用两次前向传播替代传统反向传播，但主要局限于监督学习。本文受FF中使用层活动统计量的优值函数启发，提出动作条件化的均方根Q函数（ARQ），一种用于局部强化学习的新型价值估计方法，结合优值函数与动作条件化，基于时序差分学习实现。尽管结构简单且具生物基础，该方法在MinAtar和DeepMind Control Suite基准上表现优于当前最先进的无反向传播强化学习方法，多数任务上甚至超过使用反向传播训练的算法。代码已公开于https://github.com/agentic-learning-ai-lab/arq。
原文摘要 · Abstract (English)
The Forward-Forward (FF) Algorithm is a recently proposed learning procedure for neural networks that employs two forward passes instead of the traditional forward and backward passes used in backpropagation. However, FF remains largely confined to supervised settings, leaving a gap at domains where learning signals can be yielded more naturally such as RL. In this work, inspired by FF's goodness function using layer activity statistics, we introduce Action-conditioned Root mean squared Q-Functions (ARQ), a novel value estimation method that applies a goodness function and action conditioning for local RL using temporal difference learning. Despite its simplicity and biological grounding, our approach achieves superior performance compared to state-of-the-art local backprop-free RL methods in the MinAtar and the DeepMind Control Suite benchmarks, while also outperforming algorithms trained with backpropagation on most tasks. Code can be found at https://github.com/agentic-learning-ai-lab/arq.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。