让机器人根据不确定程度自动选择执行、提问或学习，减少无效人工干预。
When to Act, Ask, or Learn: Uncertainty-Aware Policy Steering
- 结合语义不确定性和动作可行性，动态选择执行、提问或请求修正
- 在模拟和真实硬件上验证，减少40%以上的人工干预次数
- 用置信区间校准模型，支持持续学习且仅需少量人工反馈
策略引导是一种部署时自适应机器人行为的新方法：由一个预训练的策略（如扩散策略）生成低层动作样本，再由一个学习得到的验证器分析并仅选择符合任务的样本。尽管视觉语言模型（VLM）因其推理能力成为有前景的通用验证器，但现有框架通常假设其判断是准确的。实际上，当任务描述存在高层语义不确定性，或预训练策略存在低层动作不确定性或能力不足时，过度自信的VLM判断会降低引导性能。本文提出不确定性感知策略引导（UPS），联合推理语义任务不确定性和低层动作可行性，并选择合适的不确定性解决策略：执行高置信度动作、通过自然语言提问澄清任务歧义，或请求动作干预以纠正低层策略。我们利用分位数预测校准VLM与基础策略的组合，提供统计保证，确保验证器正确选择策略。部署中收集的干预数据用于残差学习，持续提升预训练策略能力，实现持续学习但仅需极少昂贵的人类反馈。我们在仿真和真实硬件上验证了该框架，结果表明UPS能有效区分确定、模糊和不可行场景，在对比未校准基线及先前的人类或机器人门控持续学习方法中显著减少人工干预。
原文摘要 · Abstract (English)
Policy steering is an emerging way to adapt robot behaviors at deployment-time: a learned verifier analyzes low-level action samples proposed by a pre-trained policy (e.g., diffusion policy) and selects only those aligned with the task. While Vision-Language Models (VLMs) are promising general-purpose verifiers due to their reasoning capabilities, existing frameworks often assume these models are well-calibrated. In practice, the overconfident judgment from VLM can degrade the steering performance under both high-level semantic uncertainty in task specifications and low-level action uncertainty or incapability of the pre-trained policy. We propose uncertainty-aware policy steering (UPS), a framework that jointly reasons about semantic task uncertainty and low-level action feasibility, and selects an uncertainty resolution strategy: execute a high-confidence action, clarify task ambiguity via natural language queries, or ask for action interventions to correct the low-level policy when it is deemed incapable at the task. We leverage conformal prediction to calibrate the composition of the VLM and the pre-trained base policy, providing statistical assurances that the verifier selects the correct strategy. After collecting interventions during deployment, we employ residual learning to improve the capability of the pre-trained policy, enabling the system to learn continually but with minimal expensive human feedback. We demonstrate our framework through experiments in simulation and on hardware, showing that UPS can disentangle confident, ambiguous, and incapable scenarios and minimizes expensive user interventions compared to uncalibrated baselines and prior human- or robot-gated continual learning approaches. Videos can be found at https://jessie-yuan.github.io/ups/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。