arXiv:2509.05368cs.ROcs.AI2025-09被引 1

通过计划与代码双重反思,提升长序列视觉模仿学习的准确性

Long-Horizon Visual Imitation Learning via Plan and Code Reflection

  • 引入计划与代码双重反思模块,动态验证和修正动作序列
  • 在300个长达18步的动作演示上,显著优于现有方法
  • 适合研究复杂时空依赖任务的视觉模仿学习者

长时序复杂动作序列的视觉模仿学习面临重大挑战,尤其在理解动作时序关系与物体空间关系方面。本文提出一种新代理框架,包含两个专用反思模块,分别用于增强计划生成与代码生成。计划生成模块产生初始动作序列,由计划反思模块验证其时序连贯性与与示范视频的空间对齐性;代码生成模块将计划转化为可执行代码,代码反思模块则验证并优化代码以确保正确性和与计划的一致性。这两个反思模块协同工作,可检测并纠正计划与代码生成中的错误,从而提升具有复杂时空依赖的任务表现。为支持系统评估,我们构建了LongVILBench基准,包含300个真人演示,动作序列最长达18步,强调多任务类型下的时序与空间复杂性。实验结果表明,现有方法在此基准上表现不佳,而本文框架建立了强基线。

原文摘要 · Abstract (English)

Learning from long-horizon demonstrations with complex action sequences presents significant challenges for visual imitation learning, particularly in understanding temporal relationships of actions and spatial relationships between objects. In this paper, we propose a new agent framework that incorporates two dedicated reflection modules to enhance both plan and code generation. The plan generation module produces an initial action sequence, which is then verified by the plan reflection module to ensure temporal coherence and spatial alignment with the demonstration video. The code generation module translates the plan into executable code, while the code reflection module verifies and refines the generated code to ensure correctness and consistency with the generated plan. These two reflection modules jointly enable the agent to detect and correct errors in both the plan generation and code generation, improving performance in tasks with intricate temporal and spatial dependencies. To support systematic evaluation, we introduce LongVILBench, a benchmark comprising 300 human demonstrations with action sequences of up to 18 steps. LongVILBench emphasizes temporal and spatial complexity across multiple task types. Experimental results demonstrate that existing methods perform poorly on this benchmark, whereas our new framework establishes a strong baseline for long-horizon visual imitation learning.

视觉模仿长时序反思机制代码生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。