给视觉语言模型加个记事本，让它能记住任务进度和物体位置。
Notes-to-Self: Scratchpad Augmented VLAs for Memory Dependent Manipulation Tasks
- 用语言记事本存储物体位置和计划步骤，实现时空记忆。
- 在多个记忆依赖任务上，模型泛化能力显著提升。
- 适合需要长期规划的机器人操作任务，尤其对非循环模型有效。
许多灵巧操作任务具有非马尔可夫特性，但当前视觉-语言-动作（VLA）模型大多忽略这一点。尽管这些模型在引入互联网级语义理解方面取得成功，但它们主要为无状态设计，难以处理依赖记忆的长时序任务。本文通过引入语言记事本，为VLA赋予空间与时间记忆能力。记事本可存储特定任务信息，如物体位置，并帮助模型跟踪计划进展与子目标达成情况。我们在ClevrSkills环境的一个记忆依赖任务子集、MemoryBench以及一项挑战性的真实世界抓取放置任务上评估该方法。结果表明，加入语言记事本显著提升了非循环与循环模型在这些任务上的泛化性能。
原文摘要 · Abstract (English)
Many dexterous manipulation tasks are non-markovian in nature, yet little attention has been paid to this fact in the recent upsurge of the vision-language-action (VLA) paradigm. Although they are successful in bringing internet-scale semantic understanding to robotics, existing VLAs are primarily "stateless" and struggle with memory-dependent long horizon tasks. In this work, we explore a way to impart both spatial and temporal memory to a VLA by incorporating a language scratchpad. The scratchpad makes it possible to memorize task-specific information, such as object positions, and it allows the model to keep track of a plan and progress towards subgoals within that plan. We evaluate this approach on a split of memory-dependent tasks from the ClevrSkills environment, on MemoryBench, as well as on a challenging real-world pick-and-place task. We show that incorporating a language scratchpad significantly improves generalization on these tasks for both non-recurrent and recurrent models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。