用手势指挥多机器人团队，智能分配任务。
GestOS: Advanced Hand Gesture Interpretation via Large Language Models to control Any Type of Robot
- 手势转文本，大模型理解意图并生成指令
- 根据机器人能力实时匹配最优执行者
- 无需指定目标，适合动态场景协同
我们提出GestOS，一种基于手势的通用操作系统，用于对异构机器人团队进行高层级控制。与以往将手势映射为固定命令或单一机器人动作的系统不同，GestOS通过语义理解手势，并依据机器人的能力、当前状态和支持的指令集，动态分配任务。系统结合轻量级视觉感知与大语言模型（LLM）推理：手部姿态被转化为结构化文本描述，由LLM推断用户意图并生成适配各机器人的具体命令。一个机器人选择模块可实时将每个手势触发的任务匹配到最合适的执行者。该架构实现了上下文感知、自适应的控制，无需用户明确指定目标或命令。GestOS将手势交互从识别提升至智能编排层次，支持在动态环境中实现可扩展、灵活且用户友好的机器人协作。
原文摘要 · Abstract (English)
We present GestOS, a gesture-based operating system for high-level control of heterogeneous robot teams. Unlike prior systems that map gestures to fixed commands or single-agent actions, GestOS interprets hand gestures semantically and dynamically distributes tasks across multiple robots based on their capabilities, current state, and supported instruction sets. The system combines lightweight visual perception with large language model (LLM) reasoning: hand poses are converted into structured textual descriptions, which the LLM uses to infer intent and generate robot-specific commands. A robot selection module ensures that each gesture-triggered task is matched to the most suitable agent in real time. This architecture enables context-aware, adaptive control without requiring explicit user specification of targets or commands. By advancing gesture interaction from recognition to intelligent orchestration, GestOS supports scalable, flexible, and user-friendly collaboration with robotic systems in dynamic environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。