深度强化学习模型自动生成类似生物计时的振荡机制。
Emergent time-keeping mechanisms in a deep reinforcement learning agent performing an interval timing task
- 通过振荡神经活动实现时间间隔的自主计时。
- 在不同视频下仍保持精准计时，不依赖外部环境。
- 与大脑基底节频率模型相似，适合研究生物时间感知。
类比深度人工神经网络与生物系统有助于理解复杂但难以解析的生物学机制。时间处理是其中一例，其内在机制尚不清晰。本研究探讨了深度强化学习（DRL）代理在执行间隔定时任务时的时间处理行为，并探索其潜在的生物学对应机制。该代理成功训练完成持续目标间隔标记任务，需在观看视频序列时识别并重复特定时间间隔。分析其内部状态发现存在振荡神经激活，这在生物系统中广泛存在。值得注意的是，代理的行为主要受高振幅、与目标间隔频率一致的振荡神经元驱动。其时间策略与生物上合理的基底节拍频率（Striatal Beat Frequency, SBF）模型高度相似。此外，代理在不同视频序列（包括空白视频）测试中仍保持振荡表示和任务性能，表明一旦习得，其时间机制已内化且对外部环境依赖极低。研究还讨论了这种涌现行为与生物过程如昼夜节律演化之间的可能关联。本研究旨在推动利用深度神经网络理解生物系统，尤其聚焦于时间处理机制。
原文摘要 · Abstract (English)
Drawing parallels between Deep Artificial Neural Networks (DNNs) and biological systems can aid in understanding complex biological mechanisms that are difficult to disentangle. Temporal processing, an extensively researched topic, is one such example that lacks a coherent understanding of its underlying mechanisms. In this study, we investigate temporal processing in a Deep Reinforcement Learning (DRL) agent performing an interval timing task and explore potential biological counterparts to its emergent behavior. The agent was successfully trained to perform a duration production task, which involved marking successive occurrences of a target interval while viewing a video sequence. Analysis of the agent's internal states revealed oscillatory neural activations, a ubiquitous pattern in biological systems. Interestingly, the agent's actions were predominantly influenced by neurons exhibiting these oscillations with high amplitudes and frequencies corresponding to the target interval. Parallels are drawn between the agent's time-keeping strategy and the Striatal Beat Frequency (SBF) model, a biologically plausible model of interval timing. Furthermore, the agent maintained its oscillatory representations and task performance when tested on different video sequences (including a blank video). Thus, once learned, the agent internalized its time-keeping mechanism and showed minimal reliance on its environment to perform the timing task. A hypothesis about the resemblance between this emergent behavior and certain aspects of the evolution of biological processes like circadian rhythms, has been discussed. This study aims to contribute to recent research efforts of utilizing DNNs to understand biological systems, with a particular emphasis on temporal processing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。