仅用1-30次未标注示范,让机器人自动学会复杂操作任务。
Build on Priors: Vision--Language--Guided Neuro-Symbolic Imitation Learning for Data-Efficient Real-World Robot Manipulation
- 用视觉语言模型自动分解动作、识别状态,构建状态转移图。
- 从单次示范生成多场景数据,提升学习效率并丰富训练集。
- 无需人工设计规则,适合工业级机器人快速部署与泛化。
让机器人从极少量示范中学习长时序操作任务仍是机器人领域的核心挑战。现有神经符号方法常依赖手工设计的符号抽象、语义标注轨迹或大规模示范数据集,限制了其可扩展性与实际应用。本文提出一种可扩展的神经符号框架,仅需1至30个未标注的动作示范,即可自动构建符号规划域和高效控制策略,无需人工领域工程。方法将示范自动分割为技能单元,并利用视觉语言模型(VLM)分类技能、识别等价高层状态,生成状态转移图;该图经答案集编程(ASP)求解器处理后,合成PDDL规划域;再通过一个预言函数,为每个技能策略提取最小、任务相关且相对目标的观测与动作空间。策略在控制参考层级而非原始执行器信号层级学习,使学习目标更平滑、噪声更少。已知控制器可用于真实世界数据增强:将单一示范投影到场景中其他物体上,同时丰富图结构与模仿学习数据集。我们在真实工业叉车平台上进行了统计严谨的操控实验验证,且在Kinova Gen3机械臂上跨平台展示了两个标准基准的通用性。结果表明,将控制学习、基于VLM的抽象与自动化规划合成统一流程,是实现可扩展、数据高效、无需专家、可解释的神经符号机器人的可行路径。
原文摘要 · Abstract (English)
Enabling robots to learn long-horizon manipulation tasks from a handful of demonstrations remains a central challenge in robotics. Existing neuro-symbolic approaches often rely on hand-crafted symbolic abstractions, semantically labeled trajectories or large demonstration datasets, limiting their scalability and real-world applicability. We present a scalable neuro-symbolic framework that autonomously constructs symbolic planning domains and data-efficient control policies from as few as one to thirty unannotated skill demonstrations, without requiring manual domain engineering. Our method segments demonstrations into skills and employs a Vision-Language Model (VLM) to classify skills and identify equivalent high-level states, enabling automatic construction of a state-transition graph. This graph is processed by an Answer Set Programming solver to synthesize a PDDL planning domain, which an oracle function exploits to isolate the minimal, task-relevant and target relative observation and action spaces for each skill policy. Policies are learned at the control reference level rather than at the raw actuator signal level, yielding a smoother and less noisy learning target. Known controllers can be leveraged for real-world data augmentation by projecting a single demonstration onto other objects in the scene, simultaneously enriching the graph construction process and the dataset for imitation learning. We validate our framework primarily on a real industrial forklift across statistically rigorous manipulation trials, and demonstrate cross-platform generality on a Kinova Gen3 robotic arm across two standard benchmarks. Our results show that grounding control learning, VLM-driven abstraction, and automated planning synthesis into a unified pipeline constitutes a practical path toward scalable, data-efficient, expert-free and interpretable neuro-symbolic robotics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。