模仿婴儿运动噪声提升强化学习探索效率
Infant Spontaneous Movement Noise Improves Exploration in Deep RL
- 用随年龄变化的彩色噪声模拟婴儿运动规律
- 在多个环境上实现更高效的学习速度
- 适合关注生物启发智能的科研人员
深度强化学习中的探索通常采用无相关性的白噪声。但近期研究表明，具有时间相关性的彩色噪声可通过生成平滑轨迹，提升状态空间覆盖效率。本文探究是否可借鉴婴儿自发运动的噪声模式来改进探索。研究发现，婴儿末端执行器速度的功率谱密度符合彩色噪声过程，且谱指数随年龄增加而上升。受此发育规律启发，我们提出一种训练过程中逐步增强探索噪声时间自相关的机制，以匹配婴儿运动统计特性。在多个强化学习环境中实验表明，这种受婴儿启发的噪声能产生结构化探索行为，显著提升学习效率。结果表明，人类运动与认知发展可为人工智能学习机制设计提供有效指导。代码已开源：https://github.com/trieschlab/baby-noise-rl。
原文摘要 · Abstract (English)
Exploration in deep reinforcement learning (RL) is commonly implemented as temporally uncorrelated white noise. However, recent works show that temporally correlated colored noise can improve exploration efficiency by producing smooth trajectories with better coverage of the state space. We inquire whether action noise inspired by infant spontaneous movements can also improve exploration in deep RL. We find that the power spectral densities of babies' end-effector velocities follow a colored noise process where the spectral exponent increases with age. Inspired by this developmental pattern, we introduce a mechanism that progressively increases the temporal auto-correlation of exploration noise during RL training, matching the infant statistics. Experiments across several RL environments show that infant-inspired noise produces structured exploratory behavior and can improve learning efficiency compared to conventional exploration strategies. These findings suggest that human motor and cognitive development can provide useful guidance for designing learning mechanisms in artificial agents. Our code is available at https://github.com/trieschlab/baby-noise-rl.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。