用可变长度物体标记实现视觉强化学习的强泛化能力
SegDAC: Visual Generalization in Reinforcement Learning via Dynamic Object Tokens
- 基于文本引导分割生成动态物体标记，无需固定槽位或重建图像
- 在8个任务上对12种视觉扰动测试，最难设置下性能提升88%
- 适合需要高视觉鲁棒性的机器人操控场景
在像素观测上训练的视觉强化学习策略在测试时若视觉条件变化，往往难以泛化。以物体为中心的表示是潜在解决方案，但多数方法依赖固定大小的槽位表示、需图像重建或辅助损失来学习物体分解。因此，如何直接从物体级输入学习强化学习策略仍不明确。我们提出SegDAC，一种基于分割驱动的演员-评论家框架，可处理可变长度的物体标记嵌入。每个时间步，通过文本引导分割生成物体掩码，并提取空间感知的标记嵌入。基于Transformer的演员-评论家网络处理这些动态标记，利用分割位置编码保留跨物体的空间信息。我们进行消融实验，证明分割位置编码和可变长度处理均对高性能至关重要。我们在8个ManiSkill3操作任务上，针对3种难度等级下的12种视觉扰动类型评估该方法。结果表明，相较于以往视觉泛化方法，其在易、中、难设置下分别提升15%、66%和88%。同时，其样本效率达到当前最先进视觉强化学习方法水平，且在视觉变化下表现更优。
原文摘要 · Abstract (English)
Visual reinforcement learning policies trained on pixel observations often struggle to generalize when visual conditions change at test time. Object-centric representations are a promising alternative, but most approaches use fixed-size slot representations, require image reconstruction, or need auxiliary losses to learn object decompositions. As a result, it remains unclear how to learn RL policies directly from object-level inputs without these constraints. We propose SegDAC, a Segmentation-Driven Actor-Critic that operates on a variable-length set of object token embeddings. At each timestep, text-grounded segmentation produces object masks from which spatially aware token embeddings are extracted. A transformer-based actor-critic processes these dynamic tokens, using segment positional encoding to preserve spatial information across objects. We ablate these design choices and show that both segment positional encoding and variable-length processing are individually necessary for strong performance. We evaluate SegDAC on 8 ManiSkill3 manipulation tasks under 12 visual perturbation types across 3 difficulty levels. SegDAC improves over prior visual generalization methods by 15% on easy, 66% on medium, and 88% on the hardest settings. SegDAC matches the sample efficiency of the state-of-the-art visual RL methods while achieving improved generalization under visual changes. Project Page: https://segdac.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。