用未来论文验证研究提案,让AI更懂科研方向。
Learning to Predict Future-Aligned Research Proposals with Language Models
- 将研究提案生成转化为预测未来论文的时序任务。
- 新指标FAS显示模型提升最高达10.6%,人类评估也认可质量提升。
- 适合想用AI辅助科研选题或创新探索的研究者。
大型语言模型在科研创意辅助中应用日益广泛,但评估其生成的研究提案质量仍具挑战:新颖性与合理性难以自动衡量,大规模人工评估成本高昂。本文提出一种可验证的替代方案,将提案生成重构为时间切片的科学预测问题。给定截止时间前的科研问题和参考文献,模型生成结构化提案,并通过其是否预测到截止时间后发表的论文来评估。我们引入未来对齐分数(FAS),基于检索和大模型语义打分计算,使用预留的未来论文库进行评估。构建了包含21,835篇论文实例、3,642个研究目标的时间一致数据集,并合成推理轨迹以训练模型识别研究空白和借鉴灵感。在Llama-3.1和Qwen2.5上,未来对齐微调使整体FAS最高提升10.6%,领域专家人工评估亦证实提案质量改善。最后,我们通过代码代理实现两个模型生成提案,在MATH数据集上新提示策略带来4.17%准确率提升,且新型模型融合方法持续改进性能。代码与数据已公开于https://github.com/Arthur-Heng/future-aligned-proposals。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly used to assist ideation in research, but evaluating the quality of LLM-generated research proposals remains difficult: novelty and soundness are hard to measure automatically, and large-scale human evaluation is costly. We propose a verifiable alternative by reframing proposal generation as a time-sliced scientific forecasting problem. Given a research question and inspiring papers available before a cutoff time, the model generates a structured proposal and is evaluated by whether it anticipates research directions that appear in papers published after the time. We operationalize this objective with the Future Alignment Score (FAS), computed via retrieval and LLM-based semantic scoring against a held-out future corpus. To train models, we build a time-consistent dataset of 21,835 paper occurrences across 3,642 instances from targets and their pre-cutoff citations, and synthesize reasoning traces that teach gap identification and inspiration borrowing. Across Llama-3.1 and Qwen2.5 models, future-aligned tuning improves future alignment over unaligned baselines (up to +10.6% overall FAS), and domain-expert human evaluation corroborates improved proposal quality. Finally, we demonstrate practical impact by implementing two model-generated proposals with a code agent, obtaining 4.17% accuracy gain on MATH from a new prompting strategy and consistent improvements for a novel model-merging method. Our code and data are publicly available at https://github.com/Arthur-Heng/future-aligned-proposals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。