揭示扩散规划中组合泛化的三大关键因素,可提升智能体生成新行为的能力。
What Do You Need for Compositional Generalization in Diffusion Planning?
- 发现平移等变性、局部感受野和推理选择是组合泛化的核心机制。
- 局部感受野比平移等变性更重要,二者共同决定扩散规划的组合能力。
- 提出Eq-Net架构,无需重规划或数据扩展即可实现高效多样性生成。
在策略学习中,拼接与组合泛化指策略利用训练数据中的子轨迹生成新且多样行为的能力。尽管离线强化学习展现出显著拼接能力,近期基于扩散规划的生成式行为克隆(BC)方法也表现出良好拼接性能,但其内在机制仍不清晰,阻碍了可设计性算法的发展。聚焦于通过生成式行为克隆训练的扩散规划器,不依赖动态规划或TD学习,我们发现三个关键属性:平移等变性、局部感受野和推理选择。这些属性解释了现有方法在架构、数据和推理上的选择,如重规划频率、数据增强和数据规模。实验表明,局部感受野比平移等变性更关键,但两者缺一不可。基于此,我们提出简单高效的扩散规划器架构Eq-Net,能生成与重规划或大规模数据扩展相当的多样化轨迹,并可在目标条件设置下引导泛化。在多类导航与操作任务中,Eq-Net展现出显著的组合泛化能力。
原文摘要 · Abstract (English)
In policy learning, stitching and compositional generalization refer to the extent to which the policy is able to piece together sub-trajectories of data it is trained on to generate new and diverse behaviours. While stitching has been identified as a significant strength of offline reinforcement learning, recent generative behavioural cloning (BC) methods have also shown proficiency at stitching. However, the main factors behind this are poorly understood, hindering the development of new algorithms that can reliably stitch by design. Focusing on diffusion planners trained via generative behavioural cloning, and without resorting to dynamic programming or TD-learning, we find three properties are key enablers for composition: shift equivariance, local receptive fields, and inference choices. We use these properties to explain architecture, data, and inference choices in existing generative BC methods based on diffusion planning including replanning frequency, data augmentation, and data scaling. Our experiments show that while local receptive fields are more important than shift equivariance in creating a diffusion planner capable of composition, both are crucial. Using findings from our experiments, we develop a new architecture for diffusion planners called Eq-Net, that is simple, produces diverse trajectories competitive with more computationally expensive methods such as replanning or scaling data, and can be guided to enable generalization in goal-conditioned settings. We show that Eq-Net exhibits significant compositional generalization in a variety of navigation and manipulation tasks designed to test planning diversity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。