让机器人扩散策略更抗错，用新方法纠正人类反馈的错误。
RoDiF: Robust Direct Fine-Tuning of Diffusion Policies with Corrupted Human Feedback
- 将扩散过程建模为统一MDP，实现无奖励直接偏好优化。
- 在30%反馈错误下仍保持性能，优于当前最佳方法。
- 适合需要高鲁棒性的长时程机器人操控任务。
扩散策略是机器人控制的强大范式，但其多步去噪过程使得基于人类偏好的微调面临根本挑战。为此，我们提出统一马尔可夫决策过程（MDP）框架，将扩散去噪链与环境动态有机结合，实现无需奖励的直接偏好优化（DPO）。在此基础上，我们提出RoDiF（Robust Direct Fine-Tuning），显式处理被污染的人类偏好。RoDiF通过几何假设切割视角重构DPO目标，并采用保守切割策略，在不假设特定噪声分布的前提下实现鲁棒性。在长时程操作任务上的大量实验表明，RoDiF持续优于现有最先进基线，能有效引导多种架构的预训练扩散策略转向人类偏好的模式，即使在30%偏好标签被污染的情况下仍保持强性能。
原文摘要 · Abstract (English)
Diffusion policies are a powerful paradigm for robotic control, but fine-tuning them with human preferences is fundamentally challenged by the multi-step structure of the denoising process. To overcome this, we introduce a Unified Markov Decision Process (MDP) formulation that coherently integrates the diffusion denoising chain with environmental dynamics, enabling reward-free Direct Preference Optimization (DPO) for diffusion policies. Building on this formulation, we propose RoDiF (Robust Direct Fine-Tuning), a method that explicitly addresses corrupted human preferences. RoDiF reinterprets the DPO objective through a geometric hypothesis-cutting perspective and employs a conservative cutting strategy to achieve robustness without assuming any specific noise distribution. Extensive experiments on long-horizon manipulation tasks show that RoDiF consistently outperforms state-of-the-art baselines, effectively steering pretrained diffusion policies of diverse architectures to human-preferred modes, while maintaining strong performance even under 30% corrupted preference labels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。