通过动态调整模型秩,实现扩散策略训练的高效与灵活。
Dynamic Rank Adjustment in Diffusion Policies for Efficient and Flexible Training
- 基于奇异值分解实现训练中动态调整可训练参数量。
- 在保持性能前提下,训练速度提升且样本效率更高。
- 适合需要快速迭代的机器人交互学习场景。
近期,通过离线行为克隆训练的扩散策略在机器人运动生成中受到关注。尽管有效,这些策略通常需要大量可训练参数,虽能提供强大表征能力,却带来高昂的训练计算成本。理想情况下,应能按需动态调整可训练部分,平衡表征力与计算效率。例如,过度参数化虽使扩散策略可通过离线行为克隆捕捉复杂机器人行为,但计算开销过大,导致在线交互式模仿学习因训练时间过长而不可行。为此,我们提出名为DRIFT的框架,利用奇异值分解(SVD)实现扩散策略训练过程中的动态秩调整。我们在DRIFT-DAgger中实现并验证该框架,这是一种可无缝切换离线预训练与在线交互阶段的模仿学习算法。通过大量实验,证明了该框架的有效性:在对模型性能影响极小的前提下,显著提升了样本效率并加快了训练速度。
原文摘要 · Abstract (English)
Diffusion policies trained via offline behavioral cloning have recently gained traction in robotic motion generation. While effective, these policies typically require a large number of trainable parameters. This model size affords powerful representations but also incurs high computational cost during training. Ideally, it would be beneficial to dynamically adjust the trainable portion as needed, balancing representational power with computational efficiency. For example, while overparameterization enables diffusion policies to capture complex robotic behaviors via offline behavioral cloning, the increased computational demand makes online interactive imitation learning impractical due to longer training time. To address this challenge, we present a framework, called DRIFT, that uses the Singular Value Decomposition to enable dynamic rank adjustment during diffusion policy training. We implement and demonstrate the benefits of this framework in DRIFT-DAgger, an imitation learning algorithm that can seamlessly slide between an offline bootstrapping phase and an online interactive phase. We perform extensive experiments to better understand the proposed framework, and demonstrate that DRIFT-DAgger achieves improved sample efficiency and faster training with minimal impact on model performance. The project website is available at: https://apollo-lab-yale.github.io/25-RSS-DRIFT-website/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。