用代码生成VLM+扩散模型，让机器人学会复杂任务的分步执行。
From Code to Action: Hierarchical Learning of Diffusion-VLM Policies
- 用视觉语言模型将任务描述拆解为可执行子程序。
- 结合扩散策略与记忆机制，实现跨时间的上下文保持与泛化。
- 支持模块化评估，适合长时序复杂操作任务研究。
机器人操作中的模仿学习常因泛化能力差和数据稀缺而受限，尤其在复杂、长时程任务中更为明显。本文提出一种分层框架，利用生成代码的视觉-语言模型（VLM）结合低层扩散策略，有效模仿并推广机器人行为。核心思想是将开源机器人API视为不仅提供执行接口，还作为结构化监督信号：当暴露子任务函数时，可作为语义明确的模块化标签。我们训练VLM将任务描述分解为可执行子程序，并通过扩散策略学习对应机器人行为。针对代码执行和某些真实任务（如物体交换）的非马尔可夫特性，架构引入记忆机制以跨时间维护子任务上下文。实验表明，该设计实现可解释的策略分解，在泛化性上优于扁平策略，且支持高层规划与低层控制的独立评估。
原文摘要 · Abstract (English)
Imitation learning for robotic manipulation often suffers from limited generalization and data scarcity, especially in complex, long-horizon tasks. In this work, we introduce a hierarchical framework that leverages code-generating vision-language models (VLMs) in combination with low-level diffusion policies to effectively imitate and generalize robotic behavior. Our key insight is to treat open-source robotic APIs not only as execution interfaces but also as sources of structured supervision: the associated subtask functions - when exposed - can serve as modular, semantically meaningful labels. We train a VLM to decompose task descriptions into executable subroutines, which are then grounded through a diffusion policy trained to imitate the corresponding robot behavior. To handle the non-Markovian nature of both code execution and certain real-world tasks, such as object swapping, our architecture incorporates a memory mechanism that maintains subtask context across time. We find that this design enables interpretable policy decomposition, improves generalization when compared to flat policies and enables separate evaluation of high-level planning and low-level control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。