用固定噪声向量替换随机噪声,显著提升生成式机器人策略性能。
You've Got a Golden Ticket: Improving Generative Robot Policies With A Single Noise Vector
- 用常量噪声替代随机初始噪声,无需训练即可优化策略。
- 51个任务中46个表现提升,仿真任务最高成功率达55%增长。
- 单个噪声向量可跨任务通用,适合部署在真实机器人系统。
当预训练的生成式机器人策略不再从高斯分布重复采样初始噪声,而是输入一个精心选择的固定初始噪声(即‘黄金票’)时,其在下游奖励下的表现可得到提升。我们提出简单的搜索方法,利用蒙特卡洛策略评估,在不训练新网络、保持预训练策略冻结的前提下,为扩散模型或流匹配策略找到最优初始噪声。该方法仅需注入初始噪声并计算稀疏任务奖励,无需额外基础设施或模型。在模拟与真实世界机器人操作基准上,46/51个任务性能提升,部分仿真任务成功率绝对提高达55%,真实任务在60次搜索内提升28%。该方法自然扩展至多任务场景,使用单一噪声向量使7个任务平均性能提升14%。此外,针对某一任务优化的黄金票也可有效提升其他相关任务表现。我们公开了基于视觉-语言-动作模型(VLA)、扩散策略和流匹配策略的预训练模型及黄金票代码库。
原文摘要 · Abstract (English)
What happens when a pretrained generative robot policy is provided a constant initial noise as input, rather than repeatedly sampling it from a Gaussian? We demonstrate that the performance of a pretrained, frozen diffusion or flow matching policy can be improved with respect to a downstream reward by swapping the sampling of initial noise from the prior distribution (typically isotropic Gaussian) with a well-chosen, constant initial noise input - a golden ticket. We propose simple search methods to find golden tickets using Monte-Carlo policy evaluation that keeps the pretrained policy frozen, does not train any new networks, and is applicable to all diffusion/flow matching policies (and therefore many VLAs). Our approach to policy improvement makes no assumptions beyond being able to inject initial noise into the policy and calculate (sparse) task rewards of episode rollouts, making it deployable with no additional infrastructure or models. Our method improves the performance of policies in 46 out of 51 tasks across simulated and real-world robot manipulation benchmarks, with absolute improvements in success rate by up to 55% for some simulated tasks, and 28% within 60 search episodes for real-world tasks. Our approach naturally extends to multi-task settings, where we improve the average performance of a VLA policy across 7 tasks by 14% using a single noise vector. Further, we find that, a golden ticket optimized for one task can also boost performance in other related tasks for the same VLA policy. We release a codebase with pretrained policies and golden tickets for simulation benchmarks using VLAs, diffusion policies, and flow matching policies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。