arXiv:2603.12868cs.RO2026-03被引 1

用强化学习微调扩散模型导航策略,提升未知环境下的安全性和泛化能力。

Beyond Imitation: Reinforcement Learning Fine-Tuning for Adaptive Diffusion Navigation Policies

  • 利用扩散模型的多轨迹采样机制,结合无需价值网络的相对策略优化方法。
  • 在未见场景中成功率提升至58.7%,路径相似度(SPL)达0.54,碰撞减少。
  • 适用于真实四足机器人和几何分布外的新环境,适合需强泛化的自主导航系统。

基于扩散模型的机器人导航策略在大规模模仿学习数据集上训练,可直接从视觉观测生成多模态轨迹,跳过传统定位-建图-规划流程,实现优异的零样本泛化。然而,其性能受限于离线数据覆盖范围,部署于未见场景时,分布偏移常导致轨迹误差累积与严重安全问题。将强化学习用于微调扩散策略面临挑战:其迭代去噪结构阻碍有效梯度反传,且额外训练价值网络计算开销大、稳定性差。为此,本文提出专为扩散导航设计的强化学习微调框架。该方法利用扩散模型固有的多轨迹采样机制,采用组相对策略优化(GRPO),通过采样轨迹间的相对优势估计避免对独立价值网络的需求。为保留预训练表征并支持适应性更新,冻结视觉编码器,仅选择性更新解码器高层与动作头,通过在线环境反馈增强安全行为。在Isaac Sim的PointGoal任务中,本方法将未见场景的成功率从52.0%提升至58.7%,SPL从0.49增至0.54,同时降低碰撞频率。额外实验表明,微调策略可零样本迁移至真实四足平台,在几何分布外环境中保持稳定性能,证明其更强的适应性与安全泛化能力。

原文摘要 · Abstract (English)

Diffusion-based robot navigation policies trained on large-scale imitation learning datasets, can generate multi-modal trajectories directly from the robot's visual observations, bypassing the traditional localization-mapping-planning pipeline and achieving strong zero-shot generalization. However, their performance remains constrained by the coverage of offline datasets, and when deployed in unseen settings, distribution shift often leads to accumulated trajectory errors and safety-critical failures. Adapting diffusion policies with reinforcement learning is challenging because their iterative denoising structure hinders effective gradient backpropagation, while also making the training of an additional value network computationally expensive and less stable. To address these issues, we propose a reinforcement learning fine-tuning framework tailored for diffusion-based navigation. The method leverages the inherent multi-trajectory sampling mechanism of diffusion models and adopts Group Relative Policy Optimization (GRPO), which estimates relative advantages across sampled trajectories without requiring a separate value network. To preserve pretrained representations while enabling adaptation, we freeze the visual encoder and selectively update the higher decoder layers and action head, enhancing safety-aware behaviors through online environmental feedback. On the PointGoal task in Isaac Sim, our approach improves the Success Rate from 52.0% to 58.7% and SPL from 0.49 to 0.54 on unseen scenes, while reducing collision frequency. Additional experiments show that the fine-tuned policy transfers zero-shot to a real quadruped platform and maintains stable performance in geometrically out-of-distribution environments, suggesting improved adaptability and safe generalization to new domains.

扩散模型强化学习导航策略零样本泛化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。