无需重训,让不同智能体的策略在视觉和任务变化下直接对接。
Mapping representations in Reinforcement Learning via Semantic Alignment for Zero-Shot Stitching
- 用语义对齐的锚点观察估计编码器间的线性变换。
- 在CarRacing中实现背景与任务变化下的零样本性能保持。
- 适合需要快速组合已有策略的动态环境应用。
深度强化学习模型在环境观测或任务需求发生微小变化时往往无法泛化,通常需昂贵的重新训练,限制了策略的复用性。本文基于语义对齐思想,提出一种零样本方法,可在不同智能体的潜在空间间建立映射,这些智能体在不同视觉和任务变体上训练过。我们学习一个变换,将一个智能体编码器的嵌入映射到另一个智能体编码器的嵌入,无需进一步微调。该方法依赖少量语义对齐的“锚点”观察,用于估计仿射或正交变换。一旦获得变换,原有控制器即可零样本地解析另一编码器的嵌入,跳过额外训练。我们在CarRacing环境中验证了该框架在视觉和任务域转移下的高性能保持能力。通过支持现有策略的模块化重组,为动态变化环境中更鲁棒、可组合的强化学习开辟了道路。
原文摘要 · Abstract (English)
Deep Reinforcement Learning (RL) models often fail to generalize when even small changes occur in the environment's observations or task requirements. Addressing these shifts typically requires costly retraining, limiting the reusability of learned policies. In this paper, we build on recent work in semantic alignment to propose a zero-shot method for mapping between latent spaces across different agents trained on different visual and task variations. Specifically, we learn a transformation that maps embeddings from one agent's encoder to another agent's encoder without further fine-tuning. Our approach relies on a small set of "anchor" observations that are semantically aligned, which we use to estimate an affine or orthogonal transform. Once the transformation is found, an existing controller trained for one domain can interpret embeddings from a different (existing) encoder in a zero-shot fashion, skipping additional trainings. We empirically demonstrate that our framework preserves high performance under visual and task domain shifts. We empirically demonstrate zero-shot stitching performance on the CarRacing environment with changing background and task. By allowing modular re-assembly of existing policies, it paves the way for more robust, compositional RL in dynamically changing environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。