arXiv:2609.09148cs.RO2026-09

用轻量代理策略在推理时精准引导通用机器人模型,实现高效任务适应。

Proxy Policy Steering

  • 训练两个轻量代理策略,通过速度空间残差动态引导基础模型。
  • 在8个真实场景任务上平均提升53%成功率,零样本任务也能有效适配。
  • 无需修改基础模型,保留其泛化能力,适合快速部署新任务。

通用机器人策略虽具备从大规模数据中学习的广泛操作先验,但将其适配至新任务仍是部署瓶颈。这需要在有限示范下激发任务特定行为,同时不损害其泛化能力。本文提出推理时自适应方法——代理策略引导(Proxy Policy Steering, PPS),通过训练两个轻量级代理策略,利用其校准的速度空间差异来引导冻结的基础采样器。参考代理建模基础策略在目标任务观测下的行为,任务代理则从参考代理初始化,捕捉任务监督下的行为变化。二者差异构成校准的速度空间残差,在每个去噪步骤中引导基础采样器。我们识别并实证验证了该残差能有效隔离任务监督引起的改变。由于基础模型始终未被修改,其广义能力(如失败恢复)得以保留。适应仅需基础模型的前向速度预测,训练轻量且无需访问基础模型参数。在8个真实世界和4个仿真操控任务上,PPS使pi 0.5基础策略平均成功率达53%绝对提升,零至一例任务亦可生效,同时保持基础模型的广泛能力。相比LoRA微调、从头训练专家、残差策略及先前推理时引导方法,表现更优。

原文摘要 · Abstract (English)

Generalist robot policies carry broad manipulation priors from large-scale data, but specializing them to a new task remains the deployment bottleneck. This requires eliciting task-specific behavior from limited demonstrations without degrading their broad capabilities. We introduce Proxy Policy Steering (PPS), an inference-time adaptation method that resolves this challenge by training two lightweight proxy policies whose calibrated velocity-space difference steers the frozen base sampler. A reference proxy models the frozen base's behavior on target-task observations, and a task proxy, initialized from the reference, captures how this behavior changes under task supervision. Their difference forms a calibrated velocity-space residual that steers the frozen base sampler at every denoising step. We identify the conditions under which this residual isolates the change induced by task supervision, and validate them empirically. Because the base is never directly modified, its broad capabilities remain available at inference, including behaviors such as recovery from failure that the demonstrations themselves do not exercise. Adaptation requires only forward velocity predictions from the base, making PPS lightweight to train and applicable even without access to the base's parameters. On 8 real-world and 4 simulation manipulation tasks, PPS lifts the state-of-the-art pi 0.5 base policy by 53% absolute success rate on average, with zero-to-one gains on tasks the base never solves, while preserving the base's broad capabilities. PPS outperforms LoRA fine-tuning, from-scratch specialists, residual policies, and prior inference-time steering methods.

机器人推理时适应轻量级强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。