arXiv:2605.27095cs.LG2026-05

用对抗性双通道方法,让小模型从专家演示中高效学习机器人控制。

Adversarial Dual On-Policy Distillation from Expressive Teacher

论文配图:Adversarial Dual On-Policy Distillation from Expressive Teacher
图 1 · 摘自论文原文
  • 设计双信号蒸馏:奖励通道引导长程探索,动作通道提供局部精准目标。
  • 在6个机器人任务上超越基线,噪声和少样本下仍保持强鲁棒性。
  • 适合缺乏在线反馈的演示学习场景,尤其适用于数据受限的机器人应用。

在具身控制中,演示学习常采用行为克隆范式,近期基于扩散或流匹配的策略通过建模多模态专家动作提升了性能。然而这些方法仍是离线监督学习:策略仅在专家状态上训练,无法获得其实际访问状态的修正信号。在线策略蒸馏(OPD)提供了自然解决方案,但标准OPD依赖固定强教师,而演示学习中无法获取。本文提出FA-OPD,一种对抗性双通道在线策略蒸馏方法:从演示中学习流匹配(FM)教师,并与轻量级MLP学生联合训练。教师在学生轨迹上提供两个互补信号:奖励通道学习状态-动作对的专家相似性目标,通过长时程策略优化驱动在线探索;动作通道在学生访问的状态处提供密集局部目标,稳定利用。两通道协同使奖励蒸馏实现超越点对点演示的泛化,动作蒸馏将探索锚定于专家行为附近。在六个机器人导航、操作和运动基准测试中,FA-OPD优于强基线,且在噪声或有限演示下表现出更强鲁棒性。

原文摘要 · Abstract (English)

Learning from demonstrations in embodied control is often cast as behavioral cloning, and recent diffusion or flow-matching policies improve this paradigm by modeling multi-modal expert actions. Yet these methods remain offline supervised learners: the policy is trained only on expert states and receives no corrective signal on the states it actually visits. On-policy distillation (OPD) offers a natural remedy, but standard OPD assumes a strong fixed teacher, which is unavailable in demonstration-only control. We propose \textbf{FA-OPD}, an \emph{adversarial dual on-policy distillation} method in which a Flow Matching (FM) teacher is learned from demonstrations and co-trained with a lightweight MLP student. The teacher provides two complementary signals on student rollouts. The reward channel learns an expert-likeness objective over state-action pairs and drives online exploration through long-horizon policy optimization. The action channel supplies dense local targets at student-visited states, stabilizing exploitation. FA-OPD couples them so that reward distillation enables generalization beyond point-wise demonstrations, while action distillation keeps exploration anchored near expert-like behavior. Across six robot navigation, manipulation, and locomotion benchmarks, FA-OPD beats strong baselines and shows much stronger robustness under noisy or limited demonstrations. Source code: https://github.com/vanzll/FA-OPD.

机器人控制策略蒸馏流匹配演示学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。