提升视觉语言动作模型长序列操作成功率,解决记忆、验证与恢复三大缺陷。
HELM: Harness-Enhanced Long-horizon Memory for Vision-Language-Action Manipulation
- 引入情景记忆、状态验证与牵引控制器,系统性修复执行循环缺陷。
- 在LIBERO-LONG上将成功率从58.4%提升至81.5%,远超扩展上下文或微调方法。
- 适合关注机器人长程任务执行与故障恢复的研究者与开发者。
视觉-语言-动作(VLA)模型在长序列操作任务中表现不佳,尽管短序列性能优异。我们发现,仅延长上下文长度无法解决该问题,根本原因在于执行循环中的三大缺陷:记忆鸿沟、验证鸿沟和恢复鸿沟。为此提出HELM,一个模型无关框架,包含三个组件:通过CLIP索引关键帧检索任务历史的情景记忆模块(EMM),基于观察、动作、子目标和记忆条件上下文预测动作失败的可学习状态验证器(SV),以及执行回滚与重规划的牵引控制器(HC)。SV是核心贡献:其表现优于规则检查与集成不确定性基线,且依赖情景记忆。在LIBERO-LONG上,HELM将成功率从OpenVLA的58.4%提升至81.5%(+23.1个百分点),而上下文窗口扩展至H=32仅带来5.4点提升,相同预算的LoRA微调仍低12.2点。HELM也在CALVIN上显著提升长程性能,并大幅增强受控扰动下的恢复成功率。消融实验与机制分析验证各组件贡献,同时发布LIBERO-Recovery作为扰动注入协议,用于评估长程操作中的故障恢复能力。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models fail systematically on long-horizon manipulation tasks despite strong short-horizon performance. We show that this failure is not resolved by extending context length alone in the current reactive execution setting; instead, it stems from three recurring execution-loop deficiencies: the memory gap, the verification gap, and the recovery gap. We present HELM, a model-agnostic framework that addresses these deficiencies with three components: an Episodic Memory Module (EMM) that retrieves key task history via CLIP-indexed keyframes, a learned State Verifier (SV) that predicts action failure before execution from observation, action, subgoal, and memory-conditioned context, and a Harness Controller (HC) that performs rollback and replanning. The SV is the core learning contribution: it consistently outperforms rule-based feasibility checks and ensemble uncertainty baselines, and its effectiveness depends critically on access to episodic memory. On LIBERO-LONG, HELM improves task success rate by 23.1 percentage points over OpenVLA (58.4% to 81.5%), while extending the context window to H=32 yields only a 5.4-point gain and same-budget LoRA adaptation remains 12.2 points below HELM. HELM also improves long-horizon performance on CALVIN and substantially boosts recovery success under controlled perturbations. Ablations and mechanism analyses isolate the contribution of each component, and we release LIBERO-Recovery as a perturbation-injection protocol for evaluating failure recovery in long-horizon manipulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。