通过调节扩散时间步，让机器人政策预训练后更高效地探索和微调。
TMRL: Diffusion Timestep-Modulated Pretraining Enables Exploration for Efficient Policy Finetuning
- 用扩散噪声平滑行为克隆，实现模仿与探索的连续过渡。
- 在真实复杂任务中，微调耗时不足一小时即成功。
- 适合需快速落地的机器人操控场景，兼容多种输入模态。
使用行为克隆（BC）预训练机器人策略时，常因动作分布过窄而缺乏下游探索能力。本文提出统一框架，通过上下文平滑预训练（CSP）向策略输入注入前向扩散噪声，构建精确模仿与广泛动作覆盖之间的连续体。随后采用时间步调制强化学习（TMRL）进行微调，使智能体在训练过程中动态调节该条件，通过控制扩散时间步实现对探索的显式调控。该方法可无缝集成任意策略输入（如状态、3D点云或图像驱动的视觉-语言-动作模型），显著提升强化学习微调的样本效率。值得注意的是，TMRL可在不到一小时内完成复杂操作任务的真实世界微调。视频与代码见 https://weirdlabuw.github.io/tmrl/。
原文摘要 · Abstract (English)
Fine-tuning pre-trained robot policies with reinforcement learning (RL) often inherits the bottlenecks introduced by pre-training with behavioral cloning (BC), which produces narrow action distributions that lack the coverage necessary for downstream exploration. We present a unified framework that enables the exploration necessary to enable efficient robot policy finetuning by bridging BC pre-training and RL fine-tuning. Our pre-training method, Context-Smoothed Pre-training (CSP), injects forward-diffusion noise into policy inputs, creating a continuum between precise imitation and broad action coverage. We then fine-tune pre-trained policies via Timestep-Modulated Reinforcement Learning (TMRL), which trains the agent to dynamically adjust this conditioning during fine-tuning by modulating the diffusion timestep, granting explicit control over exploration. Integrating seamlessly with arbitrary policy inputs, e.g., states, 3D point clouds, or image-based VLA policies, we show that TMRL improves RL fine-tuning sample efficiency. Notably, TMRL enables successful real-world fine-tuning on complex manipulation tasks in under one hour. Videos and code available at https://weirdlabuw.github.io/tmrl/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。