让小模型提前预测科研想法成败，减少盲目实验。
Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation
- 用对比评估方式训练模型判断两个研究想法谁更可能成功。
- 微调后准确率达77.1%，超过GPT-5的61.1%。
- 模型能给出可解释理由，适合科研自动化场景。
随着语言模型加速科学发现，自动生成假设和实现方案的同时，如何在不进行实验的前提下筛选大量想法成为新瓶颈。本文探究语言模型是否能在实验前预测研究想法的实证成功率。研究聚焦于‘对比性实证预测’：给定一个基准目标和两个候选想法，预测哪个更可能取得更好表现。我们基于PapersWithCode构建了包含11,488个想法对的数据集，其结果以客观指标为依据。未经微调的8B参数模型准确率仅为30%，而经过监督微调（SFT）后提升至77.1%，优于GPT-5的61.1%。通过引入可验证奖励的强化学习（RLVR），模型被训练为发现潜在推理路径，在可解释理由支持下达到71.35%准确率。消融实验与跨域测试表明，模型对表面启发式不敏感，且在时间划分和独立构建的测试集上均具鲁棒性。结果证明，计算高效的轻量级语言模型可作为有效、客观的科研想法验证工具，为自主科学发现提供可扩展路径。
原文摘要 · Abstract (English)
As language models accelerate scientific research by automating hypothesis generation and implementation, a new bottleneck emerges: evaluating and filtering hundreds of AI-generated ideas without exhaustive experimentation. We ask whether LMs can learn to forecast the empirical success of research ideas before any experiments are run. We study comparative empirical forecasting: given a benchmark-specific research goal and two candidate ideas, predict which will achieve better benchmark performance. We construct a dataset of 11,488 idea pairs grounded in objective outcomes from PapersWithCode. While off-the-shelf 8B-parameter models struggle (30% acc.), SFT dramatically boosts performance to 77.1%, outperforming GPT-5 (61.1%). By framing evaluation as a reasoning task via Reinforcement Learning with Verifiable Rewards (RLVR), we train models to discover latent reasoning paths, achieving 71.35% acc. with interpretable justifications. Through additional ablations and out-of-distribution tests, we show robustness to surface-level heuristics and transfer to both a cross-domain time-split test set and an independently constructed test set. Our results demonstrate that compute-efficient small language models can serve as effective, objective verifiers, offering a scalable path for autonomous scientific discovery.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。