用大模型从论文中自动提取流行病学模型变量,助力科研复现。
Variable Extraction for Model Recovery in Scientific Literature
- 基于大模型和规则系统对比,大模型在变量提取上表现更优。
- 人工标注数据集验证了大模型在变量识别上的高准确率。
- 适合关注自动化科研复现与模型重建的研究者使用。
全球每年学术论文产出超500万篇,人类难以跟进任何一部分科学进展。我们需要方法来导航和解析文献中的各类成果——文本、图表、代码、模型和数据集。本文评估了从流行病学研究中提取数学模型变量(如感染率(α)、恢复率(γ)、死亡率(μ))的方法。变量提取看似基础,却是从文献中恢复模型的关键步骤。提取后可用于自动建模、仿真和结果复现。我们构建了一个基准数据集,包含人工标注的变量描述与数值。基于此,提出多种基于大语言模型(LLMs)和规则的信息抽取基线方法。分析表明,大模型方案表现最佳;尽管结合规则系统略有提升,但性能跃升主要源于大模型的迁移学习与指令微调能力。本研究展示了大模型在理解科学成果及实现自动模型恢复与仿真方面的潜力。
原文摘要 · Abstract (English)
The global output of academic publications exceeds 5 million articles per year, making it difficult for humans to keep up with even a tiny fraction of scientific output. We need methods to navigate and interpret the artifacts -- texts, graphs, charts, code, models, and datasets -- that make up the literature. This paper evaluates various methods for extracting mathematical model variables from epidemiological studies, such as ``infection rate ($α$),'' ``recovery rate ($γ$),'' and ``mortality rate ($μ$).'' Variable extraction appears to be a basic task, but plays a pivotal role in recovering models from scientific literature. Once extracted, we can use these variables for automatic mathematical modeling, simulation, and replication of published results. We introduce a benchmark dataset comprising manually-annotated variable descriptions and variable values extracted from scientific papers. Based on this dataset, we present several baseline methods for variable extraction based on Large Language Models (LLMs) and rule-based information extraction systems. Our analysis shows that LLM-based solutions perform the best. Despite the incremental benefits of combining rule-based extraction outputs with LLMs, the leap in performance attributed to the transfer-learning and instruction-tuning capabilities of LLMs themselves is far more significant. This investigation demonstrates the potential of LLMs to enhance automatic comprehension of scientific artifacts and for automatic model recovery and simulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。