自动评估大模型生成生物实验协议的能力，无需人工参与。
ProtoMed-LLM: An Automatic Evaluation Framework for Large Language Models in Medical Protocol Formulation
- 用提示词驱动框架提取伪代码，实现自动化评估。
- GPT和Cohere在协议生成中表现最优，优于其他模型。
- 支持多领域扩展，适合研究自动化科研流程的团队。
自动化生成机器人可执行的科学实验协议能显著加速科研进程。大型语言模型（LLMs）在科学协议制定任务（SPFT）中表现出色，但其能力评估仍依赖人工判断。本文提出一种灵活的自动评估框架ProtoMed-LLM，通过提示目标模型与GPT-4从生物实验协议中提取仅包含预定义实验室操作的伪代码，并利用Llama-3作为评估器，以GPT-4生成的伪代码为基准进行对比。提出的基于提示的评估方法LLAM-EVAL在评估模型、材料、标准上具有高度灵活性且免费。我们评估了GPT系列、Llama、Mixtral、Gemma、Cohere和Gemini等模型，发现GPT和Cohere在科学协议制定中表现最出色。同时引入BIOPROT 2.0数据集，包含生物学协议及其对应伪代码，有助于提升和评估模型的协议生成能力。本工作可扩展至多个领域，适用于需要针对特定目标生成协议的场景。
原文摘要 · Abstract (English)
Automated generation of scientific protocols executable by robots can significantly accelerate scientific research processes. Large Language Models (LLMs) excel at Scientific Protocol Formulation Tasks (SPFT), but the evaluation of their capabilities rely on human evaluation. Here, we propose a flexible, automatic framework to evaluate LLMs' capability on SPFT: ProtoMed-LLM. This framework prompts the target model and GPT-4 to extract pseudocode from biology protocols using only predefined lab actions and evaluates the output of the target model using LLAM-EVAL, the pseudocode generated by GPT-4 serving as a baseline and Llama-3 acting as the evaluator. Our adaptable prompt-based evaluation method, LLAM-EVAL, offers significant flexibility in terms of evaluation model, material, criteria, and is free of cost. We evaluate GPT variations, Llama, Mixtral, Gemma, Cohere, and Gemini. Overall, we find that GPT and Cohere are powerful scientific protocol formulators. We also introduce BIOPROT 2.0, a dataset with biology protocols and corresponding pseudocodes, which can aid LLMs in formulation and evaluation of SPFT. Our work is extensible to assess LLMs on SPFT across various domains and other fields that require protocol generation for specific goals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。