arXiv:2603.17947cs.LGq-bio.NC2026-03

用共享嵌入向量实现快速适应,无需重训练即可应对新任务。

Unified Policy Value Decomposition for Rapid Adaptation

  • 策略与价值函数共享低维目标嵌入向量,通过乘性门控机制调节
  • 零样本估计嵌入向量,测试时仅需单次前向传播完成适应
  • 适用于需要快速迁移的复杂控制任务,如多方向行走

复杂控制系统中的快速适应仍是强化学习的核心挑战。我们提出一种框架,其中策略与价值函数共享一个低维系数向量——目标嵌入,用于捕捉任务身份,实现无需重训练表示即可立即适应新任务。预训练阶段,通过双线性演员-评论家分解联合学习结构化价值基和兼容策略基。评论家分解为 Q = sum_k G_k(g) y_k(s,a),其中 G_k(g) 是目标条件系数向量,y_k(s,a) 为学习到的价值基函数。这种乘性门控机制类似于第5层锥体神经元中的增益调制,顶层信号调节感觉响应增益而不改变其调谐特性。基于后续特征扩展至演员,由相同系数 G_k(g) 加权一组基础策略。测试时基函数冻结,G_k(g) 通过单次前向传播零样本估计,实现无需梯度更新的即时适应。我们在 MuJoCo Ant 环境中训练软演员-评论家智能体,采用多方向运动目标,要求智能体以连续目标向量指定八个方向行进。双线性结构使每个策略头专精于部分方向,共享系数层实现跨方向泛化,通过目标嵌入空间插值适应新方向。结果表明,共享低维目标嵌入为高维控制提供了通用的快速、结构化适应机制,并揭示了高效迁移在复杂强化学习系统中可能具有的生物学合理性。

原文摘要 · Abstract (English)

Rapid adaptation in complex control systems remains a central challenge in reinforcement learning. We introduce a framework in which policy and value functions share a low-dimensional coefficient vector - a goal embedding - that captures task identity and enables immediate adaptation to novel tasks without retraining representations. During pretraining, we jointly learn structured value bases and compatible policy bases through a bilinear actor-critic decomposition. The critic factorizes as Q = sum_k G_k(g) y_k(s,a), where G_k(g) is a goal-conditioned coefficient vector and y_k(s,a) are learned value basis functions. This multiplicative gating - where a context signal scales a set of state-dependent bases - is reminiscent of gain modulation observed in Layer 5 pyramidal neurons, where top-down inputs modulate the gain of sensory-driven responses without altering their tuning. Building on Successor Features, we extend the decomposition to the actor, which composes a set of primitive policies weighted by the same coefficients G_k(g). At test time the bases are frozen and G_k(g) is estimated zero-shot via a single forward pass, enabling immediate adaptation to novel tasks without any gradient update. We train a Soft Actor-Critic agent on the MuJoCo Ant environment under a multi-directional locomotion objective, requiring the agent to walk in eight directions specified as continuous goal vectors. The bilinear structure allows each policy head to specialize to a subset of directions, while the shared coefficient layer generalizes across them, accommodating novel directions by interpolating in goal embedding space. Our results suggest that shared low-dimensional goal embeddings offer a general mechanism for rapid, structured adaptation in high-dimensional control, and highlight a potentially biologically plausible principle for efficient transfer in complex reinforcement learning systems.

强化学习快速适应策略分解生物启发

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。