用新方法让导航扩散模型学会应对复杂场景和不同机器人形态。
X-NavDP: Generalizing Navigation Diffusion Policy to Novel Behavior and Embodiments with Group Q-score Reweighted Matching

- 通过行为扰动自举探索与分组Q值重加权,提升训练效率。
- 仿真环境成功率从61.20%提升至84.28%,真实场景硬ケース达65%。
- 适合做多机器人、复杂环境导航的开发者或研究者参考。
预训练导航扩散模型依赖大规模专家演示数据,这些数据通常由全局规划器生成,仅适配单一标准机器人。这限制了模型在多样机器人形态和复杂场景(如脱离死胡同或绕行长障碍)下的泛化能力,后者需基于本地观测的多样化反应行为。后训练阶段采用强化学习可提供有效解决方案,但现有扩散模型强化学习方法改善有限,原因在于扩散策略的不可计算似然导致策略梯度不稳定且探索效率低。为此,本文提出高效扩散强化学习后训练框架GQRM(Group Q-score Reweighted Matching)。该框架包含两项互补设计:(i) 保留预训练策略先验的自举探索策略,结合行为扰动;(ii) 基于状态轨迹的分组Q值归一化机制,实现高效的分数重加权匹配。通过在异构机器人形态上分布式在线强化学习训练,最终微调后的策略X-NavDP在跨形态视觉导航任务中达到当前最优表现,仿真环境下总体成功率由61.20%提升至84.28%,真实世界硬案例成功率从10%提升至65%。代码与模型已开源。
原文摘要 · Abstract (English)
Pretraining navigation diffusion policies rely on large-scale expert demonstrations. These data are typically generated by a fully-informed oracle planner suited to a single nominal robot. This limits the policy's generalization to diverse embodiments and challenging scenarios (e.g., escaping dead ends or detouring long obstacles) that demand diverse local reactive behaviors with only onboard local observations. Post-training the policy with reinforcement learning (RL) offers a principled remedy. However, previous RL for diffusion approaches lead to only marginal improvements. This is because the intractable likelihood of diffusion policies renders policy gradients unstable in addition to inefficient policy exploration. To address these challenges, we propose a data-efficient diffusion RL post-training framework - GQRM (Group Q-score Reweighted Matching). Our framework introduces two complementary designs: (i) a self-bootstrapped exploration strategy with behavior perturbation that preserves the pretrained policy prior, and (ii) a group Q-score normalization mechanism that computes per-trajectory values on each state for efficient reweighted score matching. By conducting distributed online RL training across heterogeneous embodiments, the resulting fine-tuned policy, X-NavDP, achieves state-of-the-art cross-embodiment visual navigation performance, improving the overall success rate from 61.20% to 84.28% in simulation and 10% to 65% in real-world hard cases. The code and model are publicly available at https://yty-sky.github.io/x-navdp-project-page.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。