受昆虫启发的模块化架构提升强化学习多任务表现
Insect-inspired modular architectures as inductive biases for reinforcement learning
- 采用感官编码、方向表征等模块化设计，通过可学习仲裁机制分配控制权
- 在导航任务中平均回报达-2798.8，优于中央化GRU和MLP模型
- 模块间分工明确，控制分配高度选择性，适合复杂动态任务
大多数连续控制中的强化学习控制器采用集中式架构：观测信息压缩为单一潜在状态，由此生成价值估计与动作。而生物控制系统常采用分布式结构。以昆虫为例，其通过分散电路协调导航、方向稳定、记忆及情境化动作选择，而非单一整体控制器。受此启发，本文提出一种将控制分解为感知编码、方向表征、稀疏关联记忆、递归指令生成与局部运动控制等交互模块的策略架构，并引入可学习的仲裁机制分配运动控制权。模型在二维导航任务中评估，需同时实现觅食、避障与逃逸。在6组种子的猎物-导航实验中，使用近端策略优化（PPO）训练75轮后，模块化策略取得最佳平均性能，最终回合回报为$-2798.8\pm964.4$，显著优于集中式门控循环单元（GRU）的$-3778.0\pm628.1$和多层感知机（MLP）的$-4727.5\pm772.5$。模块化策略还实现最低的值函数损失与稳定的优化统计，且模块分配熵降至$0.0457\pm0.0244$，表明控制分配高度选择性。结果表明，分布式控制可作为处理动态竞争行为目标的有益归纳偏置。
原文摘要 · Abstract (English)
Most reinforcement-learning (RL) controllers used in continuous control are architecturally centralized: observations are compressed into a single latent state from which both value estimates and actions are produced. Biological control systems are often organized differently. Insects, in particular, coordinate navigation, heading stabilization, memory, and context-dependent action selection through distributed circuits rather than a single monolithic controller. Motivated by this contrast, we study an RL policy architecture that decomposes control into interacting modules for sensory encoding, heading representation, sparse associative memory, recurrent command generation, and local motor control, with a learned arbitration mechanism that allocates motor authority across modules. The model is evaluated on a two-dimensional navigation task that require simultaneous food seeking, obstacle avoidance, and predator escape. In a six-seed predator-navigation experiment trained with Proximal Policy Optimization (PPO) for 75 updates, the modular policy achieves the strongest final mean performance among the tested controllers, with final episodic return $-2798.8\pm964.4$ versus $-3778.0\pm628.1$ for a centralized gated recurrent unit (GRU) and $-4727.5\pm772.5$ for a centralized multilayer perceptron (MLP). The modular policy also attains the lowest final value loss and stable PPO optimization statistics while driving module-assignment entropy to $0.0457\pm0.0244$, indicating highly selective control allocation. These results suggest that distributed control can serve as a useful inductive bias for RL problems involving dynamically competing behavioral objectives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。