让机器人视觉语言动作模型自适应调整计算量,难任务多算,易任务少算。
Elastic Queries Reinforcement Learning: Self-Aware Policy Execution for VLA Models

- 根据任务难度动态调整推理次数和动作执行长度
- 实测在仿真和真实机器人上降低平均推理成本
- 无需微调模型,适合部署于资源受限的机器人系统
视觉-语言-动作(VLA)模型虽能生成强大机器人操作指令,但通常采用固定的推理与重规划周期。这种僵化处理忽略了机器人控制任务的难度差异:接触密集或不确定性高的状态需要更多计算和更及时反馈,而简单状态则可用较少推理步骤和较长开环执行完成。本文提出弹性查询强化学习(EQRL),使每个VLA策略查询具备弹性。一个轻量级潜在调度适配器联合选择潜在输入、去噪预算和动作块长度,无需微调底层VLA模型。为实现难度感知调度,EQRL在联合潜在-调度动作上训练一个价值网络,并通过价值网络集合的分歧度推导出状态难度信号,引导计算资源聚焦于困难状态;同时引入学习到的残差项实现任务驱动修正。将可变块执行建模为依赖块的宏观动作强化学习,结合块相关的折扣因子与摊销函数评估次数(NFE)预算。在仿真与真实机器人操作任务中,EQRL在保持或提升任务成功率的同时,降低了平均推理成本。
原文摘要 · Abstract (English)
Vision-language-action (VLA) models are powerful action generators for robot manipulation, but they are typically executed with fixed inference and replanning schedules. This rigidity ignores the uneven difficulty of robot control: contact-rich or uncertain states may need more computation and fresher feedback, while easier states can often be handled with fewer inference steps and longer open-loop execution. We propose Elastic Queries Reinforcement Learning (EQRL), a framework that makes each VLA policy query elastic. A lightweight latent-schedule adaptor jointly selects the latent input, denoising budget, and action chunk length, without fine-tuning the underlying VLA model. To make scheduling difficulty-aware, EQRL trains a critic over the joint latent-schedule action and derives a state difficulty signal from critic ensemble disagreement. This signal guides compute toward difficult states, while a learned residual allows task-driven correction. We formulate variable chunk execution as query-level macro-action RL with chunk-dependent discounting and an amortized number-of-function-evaluations (NFE) budget. Across simulation and real-robot manipulation, EQRL reduces amortized inference cost while preserving or improving task success.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。