用知识图谱和触觉信息实现机器人技能跨场景迁移。
Semantic-Geometric-Physical-Driven Robot Manipulation Skill Transfer via Skill Library and Tactile Representation
- 构建任务、场景、状态三重图谱，组织操作知识
- 结合大模型与触觉反馈，实现跨场景技能迁移
- 适合研究机器人通用操作与人机协作的开发者
开发能够在非结构化环境中进行操作的通用机器人系统是一项重大挑战，尤其当任务具有长时程、高接触特性时，需在不同任务场景间高效迁移技能。为此，我们提出基于知识图谱的技能库构建方法，通过任务图和场景图分别表示任务特异性和场景特异性信息，并引入状态图实现高层任务规划与底层场景信息的交互。在此基础上，我们进一步提出一种基于技能库与触觉表征的分层技能迁移框架，融合高层推理与低层执行精度。在任务层面，利用大语言模型（LLMs）结合上下文学习与四阶段思维链提示策略，实现子任务序列迁移；在运动层面，设计基于技能库与启发式路径规划的自适应轨迹迁移方法；在物理层面，提出基于触觉表征的自适应轮廓提取与姿态感知方法，从视觉-触觉图像中动态获取高精度轮廓与姿态信息，调整接触位置与姿态参数，确保技能在新环境中的有效性。实验验证了所提方法在不同任务场景下的技能迁移与适应能力。项目主页：https://github.com/MingchaoQi/skill_transfer
原文摘要 · Abstract (English)
Developing general robotic systems capable of manipulating in unstructured environments is a significant challenge, particularly as the tasks involved are typically long-horizon and rich-contact, requiring efficient skill transfer across different task scenarios. To address these challenges, we propose knowledge graph-based skill library construction method. This method hierarchically organizes manipulation knowledge using "task graph" and "scene graph" to represent task-specific and scene-specific information, respectively. Additionally, we introduce "state graph" to facilitate the interaction between high-level task planning and low-level scene information. Building upon this foundation, we further propose a novel hierarchical skill transfer framework based on the skill library and tactile representation, which integrates high-level reasoning for skill transfer and low-level precision for execution. At the task level, we utilize large language models (LLMs) and combine contextual learning with a four-stage chain-of-thought prompting paradigm to achieve subtask sequence transfer. At the motion level, we develop an adaptive trajectory transfer method based on the skill library and the heuristic path planning algorithm. At the physical level, we propose an adaptive contour extraction and posture perception method based on tactile representation. This method dynamically acquires high-precision contour and posture information from visual-tactile images, adjusting parameters such as contact position and posture to ensure the effectiveness of transferred skills in new environments. Experiments demonstrate the skill transfer and adaptability capabilities of the proposed methods across different task scenarios. Project website: https://github.com/MingchaoQi/skill_transfer
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。