用真实生物通路数据测试大模型推理能力，发现其在扰动系统中表现差，提出交互式导航方法提升性能。
BioMaze: Benchmarking and Enhancing Large Language Models for Biological Pathway Reasoning
- 构建5.1K个真实研究来源的通路推理题，覆盖动态变化与干预条件。
- 大模型在扰动系统中推理准确率低，传统提示法效果有限。
- 提出PathSeeker代理，通过子图交互导航提升科学推理能力，适合生物信息学研究者。
大语言模型（LLMs）在生物学领域的应用日益广泛，但其在复杂生物系统（如通路）中的推理能力仍缺乏深入探索，而这一能力对预测生物现象、提出假设和设计实验至关重要。本文探讨了LLMs在通路推理中的潜力，提出了BioMaze数据集，包含5.1K个源自真实研究的复杂通路问题，涵盖自然动态变化、干扰、额外干预条件及多尺度研究目标等多种生物情境。我们评估了CoT和图增强推理等方法，发现LLMs在扰动系统中表现不佳。为此，提出PathSeeker——一种通过交互式子图导航增强推理的LLM代理，实现更符合科学逻辑的复杂生物系统处理。数据集与代码已公开于https://github.com/zhao-ht/BioMaze。
原文摘要 · Abstract (English)
The applications of large language models (LLMs) in various biological domains have been explored recently, but their reasoning ability in complex biological systems, such as pathways, remains underexplored, which is crucial for predicting biological phenomena, formulating hypotheses, and designing experiments. This work explores the potential of LLMs in pathway reasoning. We introduce BioMaze, a dataset with 5.1K complex pathway problems derived from real research, covering various biological contexts including natural dynamic changes, disturbances, additional intervention conditions, and multi-scale research targets. Our evaluation of methods such as CoT and graph-augmented reasoning, shows that LLMs struggle with pathway reasoning, especially in perturbed systems. To address this, we propose PathSeeker, an LLM agent that enhances reasoning through interactive subgraph-based navigation, enabling a more effective approach to handling the complexities of biological systems in a scientifically aligned manner. The dataset and code are available at https://github.com/zhao-ht/BioMaze.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。