将生物实验流程转化为机器人可执行动作序列
ProtoAct: Turning Wet-Lab Protocols into Embodied Robotic Actions

- 用检索+校验+结构化映射三步法解析实验流程
- 在7个大模型上验证有效,真实机器人成功执行
- 适合自动化实验与机器人科研人员使用
生物湿实验流程通常针对训练有素的研究人员编写,隐含常规操作、状态依赖条件和上下文参数,难以直接转为机器人可执行动作。我们提出ProtoAct,一种结构化的协议对齐框架,将自由格式的生物学流程转化为状态感知、可具身化的动作序列。ProtoAct利用ProtoRAG检索人工标注的示例进行上下文敏感解析,采用RefineChecker检测并修正缺失或不一致步骤,并通过ActSchema将优化后的流程映射为受约束的JSON函数序列。我们进一步构建了BioP2E数据集,手动标注22个细胞培养协议,包含258个监测条件、910个可执行子任务和962个具身动作调用。在七个大语言模型上的评估表明,ProtoAct可有效适配不同模型。消融实验确认检索、后验校验与模式约束各有贡献。解析出的子任务还支持示范数据收集与视觉-语言-动作模型训练,实现在仿真与真实机器人环境中的成功执行。ProtoAct因此为生物实验理解与具身机器人执行之间提供了实用接口。
原文摘要 · Abstract (English)
Biological wet-lab protocols are written for trained researchers and often leave routine operations, state-dependent conditions, and contextual parameters implicit, making them difficult to translate into robot-executable actions. We present ProtoAct, a structured protocol-grounding framework that converts free-form biological procedures into state-aware, embodiment-ready action sequences. ProtoAct uses ProtoRAG to retrieve manually annotated examples for context-sensitive parsing, employs RefineChecker to detect and revise missing or inconsistent steps, and applies ActSchema to map the refined procedure into constrained JSON function sequences. We further introduce BioP2E, for which we manually annotate 22 cell-culture protocols into 258 monitoring conditions, 910 executable subtasks, and 962 grounded action calls. Evaluation across seven large language models demonstrates that ProtoAct can be effectively instantiated with different backbones. Ablations confirm that retrieval, posterior checking, and schema constraints make complementary contributions. The parsed subtasks further support demonstration collection and VLA model training, enabling successful execution in both simulation and real-robot settings. ProtoAct thus provides a practical interface between biological protocol understanding and embodied robotic execution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。