用专家讨论训练大模型，提升基因组学推理能力
Toward Scientific Reasoning in LLMs: Training from Expert Discussions via Reinforcement Learning
- 从科学论坛对话中构建强化学习数据集
- 推理准确率提升超15%，接近专家水平
- 适用于生物等领域的科学推理训练
我们研究如何利用专家讨论作为学习信号，教会大语言模型进行科学推理。聚焦基因组学领域，我们开发了一个自动化管道，从超过十年的基因组工程论坛讨论中提取可训练数据，并构建了Genome-Bench基准。该管道将原始互动转化为适合强化学习的多选题格式，包含3000多个高质量问答对，涵盖基础生物学、实验故障排查、工具使用等。我们使用基于规则的奖励信号，基于合成的多选题数据集对LLM进行强化学习微调，以增强领域特定推理能力。结果表明，与基础模型相比，该方法在Genome-Bench上使模型性能提升超过15%，缩小了开源LLM与专家级推理之间的差距。据我们所知，这是首个端到端的从科学讨论中教学推理的管道，具有跨科学领域推广的潜力。
原文摘要 · Abstract (English)
We investigate how to teach large language models (LLMs) to perform scientific reasoning by leveraging expert discussions as a learning signal. Focusing on the genomics domain, we develop an automated pipeline to extract trainable data and introduce Genome-Bench, a new benchmark constructed from over a decade of scientific forum discussions on genome engineering. Our pipeline transforms raw interactions into a reinforcement learning-friendly multiple-choice questions format, supported by 3000+ high-quality question-answer pairs spanning foundational biology, experimental troubleshooting, tool usage, and beyond. We fine-tune an LLM using RL with a rule-based reward signal derived from the synthetic MCQ dataset to enhance domain-specific reasoning. Our results show that reinforcement learning from scientific discussions improves model performance by over 15% compared to the base model on Genome-Bench, narrowing the gap between open-source LLMs and expert-level reasoning. To our knowledge, this is the first end-to-end pipeline for teaching LLMs to reason from scientific discussions, with promising potential for generalization across scientific domains beyond biology.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。