通过调控系统动态性,实现无需外部噪声的自适应探索。
Dynamic Reinforcement Learning for Actors
- 用神经元敏感度控制全局动态,替代传统对动作网络的直接调整。
- 在两个动态任务中无需探索噪声即可稳定训练,且适应新环境能力强。
- 为智能体“思考”能力的涌现提供新思路,适合研究自主学习与创造力的学者。
本文提出动态强化学习(Dynamic RL),不直接控制动作网络输出,而是实时调节系统动态行为,实现从静态到动态的范式转变。代理通过与环境的循环交互生成混沌动态,实现灵活而确定性的探索。该方法利用局部敏感度指标调控全局系统动态:敏感度调整学习(SAL)防止动态过度收敛;敏感度控制强化学习(SRL)则根据时序差分误差正负分别引导动态收敛或发散——正误差处收敛以增强优质状态转移的可重复性,负误差处发散以促进劣质状态的探索。该方法仅应用于演员-评论家架构中的演员,未扩展至评论家。在两个动态任务中验证有效,无需外部探索噪声或时间反向传播。展现出优异的新环境适应能力,但仍存挑战。作者类比‘探索’与‘思考’,提出‘探索通过学习成长为思考’的假设,认为该方法或为思维(包括不可复现的灵感)涌现的关键技术。最后,尽管冒昧,作者主张该研究应暂停,以引发对潜在致命风险的讨论。
原文摘要 · Abstract (English)
Dynamic Reinforcement Learning (Dynamic RL), proposed in this paper, directly controls system dynamics, instead of the actor (action-generating neural network) outputs at each moment, bringing about a major qualitative shift in reinforcement learning (RL) from static to dynamic. The actor is initially designed to generate chaotic dynamics through the loop with its environment, enabling the agent to perform flexible and deterministic exploration. Dynamic RL controls global system dynamics using a local index called "sensitivity," which indicates how much the input neighborhood contracts or expands into the corresponding output neighborhood through each neuron's processing. While sensitivity adjustment learning (SAL) prevents excessive convergence of the dynamics, sensitivity-controlled reinforcement learning (SRL) adjusts them -- to converge more to improve reproducibility around better state transitions with positive TD error and to diverge more to enhance exploration around worse transitions with negative TD error. Dynamic RL was applied only to the actor in an Actor-Critic RL architecture while applying it to the critic remains a challenge. It was tested on two dynamic tasks and functioned effectively without external exploration noise or backward computation through time. Moreover, it exhibited excellent adaptability to new environments, although some problems remain. Drawing parallels between 'exploration' and 'thinking,' the author hypothesizes that "exploration grows into thinking through learning" and believes this RL could be a key technique for the emergence of thinking, including inspiration that cannot be reconstructed from massive existing text data. Finally, despite being presumptuous, the author presents the argument that this research should not proceed due to its potentially fatal risks, aiming to encourage discussion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。