人机协作机器人通过对话理解指令，实现精准抓取与放置。
A Conversational Framework for Human-Robot Collaborative Manipulation with Distributed Generative AI models
- 将语言与视觉模型分布式部署于ROS 2系统，分步执行指令理解、视觉定位与动作规划。
- 在场景模糊度增加时仍保持95%以上任务成功率，响应延迟低于1.2秒。
- 适合需要安全交互的工业协作场景，支持可视化确认操作流程。
本文提出一种基于分布式对话框架的人机协同操作方法，将本地语言模型与视觉语言模型（VLM）集成至基于ROS 2的执行栈中。语言理解、视觉定位、任务编排与运动执行分别以独立ROS 2节点运行，可在分布式硬件上灵活部署，同时保证响应控制回路的实时性。用户输入自由文本指令后，系统生成结构化动作请求，用于抓取、放置与交接。通过VLM返回图像空间目标，结合深度信息与标定参数转换为机器人坐标系下的精确目标点。配套网页仪表盘展示中间意图与视觉定位叠加结果（像素、深度、机器人帧），并在执行前需操作员明确确认。在Franka FR3平台上开展实验，评估了在不断增大的工作台场景模糊度下端到端任务可靠性与延迟，并对比了不同LLM/VLM配置在同一流程中的表现。代码与完整文档见[github.com/cogrob-tuni/franka-llm]。
原文摘要 · Abstract (English)
This paper presents a distributed conversational framework for human-robot collaborative manipulation that integrates local language and vision-language models (VLMs) with a Robot Operating System 2 (ROS 2)-based execution stack. Language understanding, visual grounding, orchestration, and motion execution run as separate ROS 2 nodes, enabling flexible deployment across distributed hardware while maintaining a responsive control loop. From free-form user commands, the system generates structured action requests for pick, place, and handover. It uses a VLM to return image-space targets, which are converted into metric robot-frame goals using depth and calibration. A web dashboard exposes intermediate intent and grounding overlays (pixel, depth, and robot-frame) and requires explicit operator confirmation before any motion is executed. Experiments on a Franka FR3 platform evaluate end-to-end task reliability and latency under increasing working table scene ambiguity and compare alternative LLM/VLM configurations in the same pipeline. Code and full documentation are available at [github.com/cogrob-tuni/franka-llm](https://github.com/cogrob-tuni/franka-llm).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。