arXiv:2608.15549cs.ROcs.AI2026-08中稿 · ECCV

让机器人听懂自然语言指令,自动组合技能完成社交任务。

MistyPilot: Enabling Social-Robot Control through Multi-Agent LLM Skill Orchestration

论文配图:MistyPilot: Enabling Social-Robot Control through Multi-Agent LLM Skill Orchestration
图 1 · 摘自论文原文
  • 用两个专用智能体分工协作:一个管物理动作,一个管对话状态。
  • 可支持最多100个技能,任务成功率高且执行更稳定。
  • 适合想快速开发社交机器人的开发者和研究者使用。

从自然语言指令编程小型社交机器人,不能仅调用孤立的API。交互任务需结合反应式物理行为与状态化社交行为,而现有接口常要求开发者手动编排API、配置参数、绑定传感器事件并管理运行时任务状态。本文提出MistyPilot,一种多智能体大模型框架,能解析高层自然语言指令,并在Misty社交机器人上协调对应技能。任务调度器将指令分发至两个专精智能体:物理交互智能体负责传感器触发的机器人控制与直接技能调用,社交互动智能体则负责对话导向的任务状态管理及上下文相关的多模态响应生成。为提升效率,社交互动智能体在适用时复用先前结果,否则调用完整生成。我们在五个组件级测试套件上评估MistyPilot,传感器绑定与技能调用均在真实Misty机器人上执行,并开展包含12名参与者的初步用户研究。结果表明,MistyPilot在任务路由、传感器-技能绑定、任务状态解析、结果复用和技能扩展(最高达100项)上均表现优异,方差低于同等单智能体基线,用户对可用性与交互质量评价积极。代码将通过项目页面公开。

原文摘要 · Abstract (English)

Programming small social robots from natural-language instructions requires more than invoking isolated APIs. Interactive tasks combine reactive physical behaviors with stateful social behaviors, while existing interfaces often require developers to manually compose APIs into skills, configure their parameters, bind sensor events to skills, and manage task states at runtime. We present MistyPilot, a multi-agent LLM framework that interprets high-level natural-language instructions and orchestrates the corresponding skills on the Misty social robot. A Task Router dispatches each instruction to one of two specialized agents: a Physically Interactive Agent for sensor-triggered robot control and direct skill invocation, and a Social Interaction Agent for dialogue-oriented task-state management and context-dependent multimodal response generation. To improve efficiency, the Social Interaction Agent reuses previously generated results when applicable and invokes full generation otherwise. We evaluate MistyPilot on five component-level suites, with sensor bindings and skill invocations executed on the physical Misty robot, and a preliminary user study with 12 participants. MistyPilot attains high accuracy on routing, sensor-skill binding, task-state parsing, result reuse, and skill extension up to 100 skills, and lower variance than an otherwise identical single-agent baseline, while participants report positive perceptions of usability and interaction quality. The code will be made publicly available via the project page.

社交机器人多智能体大模型自然语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。