arXiv:2605.02600cs.ROcs.AI2026-05中稿 · Robotics: Science …

用大模型设计控制目标,让机器人在复杂接触场景中自主完成高难度操作。

CoRAL: Contact-Rich Adaptive LLM-based Control for Robotic Manipulation

论文配图:CoRAL: Contact-Rich Adaptive LLM-based Control for Robotic Manipulation
图 1 · 摘自论文原文
  • 大模型不直接控制,而是生成动态成本函数供采样规划器使用。
  • 通过在线系统识别实时优化物理参数,提升对摩擦、质量等的感知精度。
  • 支持复用成功策略,适合需要反复交互的复杂操控任务。

尽管大语言模型(LLMs)和视觉语言模型(VLMs)在高层推理与语义理解方面表现出色,但将其直接应用于富含接触的操纵任务仍面临缺乏显式物理建模与自适应控制能力的挑战。为此,我们提出 CoRAL(Contact-Rich Adaptive LLM-based control),一个模块化框架,实现零样本规划,通过解耦高层推理与底层控制。不同于黑箱策略,CoRAL 将 LLM 用作成本函数设计者,为基于采样的运动规划器(MPPI)生成上下文感知的目标函数。为应对视觉数据中物理参数的模糊性,我们引入神经符号适应循环:VLM 提供环境动力学的语义先验(如质量、摩擦估计),并通过在线系统识别实时修正;同时 LLM 根据交互反馈迭代调整成本函数结构以纠正策略错误。此外,基于检索的记忆单元使系统可复用过往成功策略。该分层架构通过解耦高层语义推理与实时执行,确保控制稳定性,有效弥合慢速大模型推理与动态接触需求之间的差距。我们在仿真与真实硬件上验证了 CoRAL,涵盖翻转物体撞墙等高难度新颖任务。实验表明,其在未见的接触丰富场景中平均成功率较现有 VLA 和基础模型规划器基准提升超 50%,并通过自适应物理理解有效缓解模拟到现实的差距。

原文摘要 · Abstract (English)

While Large Language Models (LLMs) and Vision-Language Models (VLMs) demonstrate remarkable capabilities in high-level reasoning and semantic understanding, applying them directly to contact-rich manipulation remains a challenge due to their lack of explicit physical grounding and inability to perform adaptive control. To bridge this gap, we propose CoRAL (Contact-Rich Adaptive LLM-based control), a modular framework that enables zero-shot planning by decoupling high-level reasoning from low-level control. Unlike black-box policies, CoRAL uses LLMs not as direct controllers, but as cost designers that synthesize context-aware objective functions for a sampling-based motion planner (MPPI). To address the ambiguity of physical parameters in visual data, we introduce a neuro-symbolic adaptation loop: a VLM provides semantic priors for environmental dynamics, such as mass and friction estimates, which are then explicitly refined in real time via online system identification, while the LLM iteratively modulates the cost-function structure to correct strategic errors based on interaction feedback. Furthermore, a retrieval-based memory unit allows the system to reuse successful strategies across recurrent tasks. This hierarchical architecture ensures real-time control stability by decoupling high-level semantic reasoning from reactive execution, effectively bridging the gap between slow LLM inference and dynamic contact requirements. We validate CoRAL on both simulation and real-world hardware across challenging and novel tasks, such as flipping objects against walls by leveraging extrinsic contacts. Experiments demonstrate that CoRAL outperforms state-of-the-art VLA and foundation-model-based planner baselines by boosting success rates over 50% on average in unseen contact-rich scenarios, effectively handling sim-to-real gaps through its adaptive physical understanding.

机器人操控大模型应用自适应控制多模态学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。