arXiv:2511.02239cs.ROcs.AI2025-11中稿 · ICRA

让机器人既能执行指令又能解释动作,实现自我改进的智能操控

LACY: A Vision-Language Model-based Language-Action Cycle for Self-Improving Robotic Manipulation

  • 用统一视觉语言模型实现语言到动作与动作到语言的双向映射
  • 在仿真和真实场景中任务成功率平均提升56.46%
  • 无需人工标注,自动筛选低置信度案例生成新训练数据

机器人泛化操控策略越来越依赖将语言指令映射为动作(L2A)的大规模模型。然而,这种单向范式常导致策略缺乏上下文理解,限制了泛化与行为解释能力。我们主张,将动作反向映射为语言(A2L)同样关键,可构建更完整的语义基础。能同时行动与解释的智能体可形成更丰富的内部表征,开启自监督学习新范式。我们提出LACY(语言-动作循环),一个在单一视觉语言模型中联合学习双向映射的统一框架。该框架在三个协同任务上联合训练:从语言生成参数化动作(L2A)、用语言解释观测动作(A2L)、验证两段语言描述的语义一致性(L2C)。这实现了自主生成与过滤新训练数据的自我改进循环,通过主动增强策略聚焦低置信度案例,从而无需额外人工标注即可持续优化模型。在仿真与真实世界的抓取放置任务实验中,LACY平均提升任务成功率56.46%,并显著增强语言-动作对齐的鲁棒性。

原文摘要 · Abstract (English)

Learning generalizable policies for robotic manipulation increasingly relies on large-scale models that map language instructions to actions (L2A). However, this one-way paradigm often produces policies that execute tasks without deeper contextual understanding, limiting their ability to generalize or explain their behavior. We argue that the complementary skill of mapping actions back to language (A2L) is essential for developing more holistic grounding. An agent capable of both acting and explaining its actions can form richer internal representations and unlock new paradigms for self-supervised learning. We introduce LACY (Language-Action Cycle), a unified framework that learns such bidirectional mappings within a single vision-language model. LACY is jointly trained on three synergistic tasks: generating parameterized actions from language (L2A), explaining observed actions in language (A2L), and verifying semantic consistency between two language descriptions (L2C). This enables a self-improving cycle that autonomously generates and filters new training data through an active augmentation strategy targeting low-confidence cases, thereby improving the model without additional human labels. Experiments on pick-and-place tasks in both simulation and the real world show that LACY improves task success rates by 56.46% on average and yields more robust language-action grounding for robotic manipulation. Project page: https://vla2026.github.io/LACY/

机器人操控多模态学习自进化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。