arXiv:2602.09940cs.ROcs.AI2026-02被引 1

让机器人听懂人话并执行复杂操作,全程本地运行无延迟。

Instruct2Act: From Human Instruction to Actions Sequencing and Execution via Robot Action Network for Robotic Manipulation

  • 用轻量模型将自然语言指令拆解为动作序列
  • 真实机器人测试成功率达90%,单步推理<3.8秒
  • 适合边缘设备部署,无需云端支持

机器人在真实场景中常因计算与感知限制难以理解自由形式的人类指令。本文提出一种轻量级、完全本地运行的流水线,将自然语言命令转化为可靠的操作行为。系统分两阶段:(i) 指令转动作模块(Instruct2Act),采用小型BiLSTM结合多头注意力自编码器,将指令解析为原子动作序列(如抓取、移动、放置);(ii) 机器人动作网络(RAN),融合动态自适应轨迹径向网络(DATRN)与基于视觉的环境分析器(YOLOv8),为每个子动作生成精确控制轨迹。整个系统运行于普通硬件,无需云服务。在自建私有数据集上,Instruct2Act实现91.5%的子动作预测准确率;四类真实任务(拾取放置、拾取倾倒、擦拭、拾取交付)的综合成功率达90%;子动作推断耗时<3.8秒,端到端执行时间30-60秒,取决于任务复杂度。结果表明,细粒度指令解析结合DATRN轨迹生成与视觉引导,为资源受限、单摄像头环境下确定性实时操作提供了可行路径。

原文摘要 · Abstract (English)

Robots often struggle to follow free-form human instructions in real-world settings due to computational and sensing limitations. We address this gap with a lightweight, fully on-device pipeline that converts natural-language commands into reliable manipulation. Our approach has two stages: (i) the instruction to actions module (Instruct2Act), a compact BiLSTM with a multi-head-attention autoencoder that parses an instruction into an ordered sequence of atomic actions (e.g., reach, grasp, move, place); and (ii) the robot action network (RAN), which uses the dynamic adaptive trajectory radial network (DATRN) together with a vision-based environment analyzer (YOLOv8) to generate precise control trajectories for each sub-action. The entire system runs on a modest system with no cloud services. On our custom proprietary dataset, Instruct2Act attains 91.5% sub-actions prediction accuracy while retaining a small footprint. Real-robot evaluations across four tasks (pick-place, pick-pour, wipe, and pick-give) yield an overall 90% success; sub-action inference completes in < 3.8s, with end-to-end executions in 30-60s depending on task complexity. These results demonstrate that fine-grained instruction-to-action parsing, coupled with DATRN-based trajectory generation and vision-guided grounding, provides a practical path to deterministic, real-time manipulation in resource-constrained, single-camera settings.

机器人操作自然语言理解本地部署动作规划

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。