用人类与算法共交易,预测科研结果能否复现。
Human-AI Collaboration for Estimating Scientific Replicability

- 人机共同参与预测市场,实时交易估复现概率。
- 混合模式在多数实验中优于纯人工或纯算法。
- 结合领域知识与历史数据,提升预测可靠性。
判断已发表科学发现是否可成功复现,是实证科学中的长期挑战。现有评估方法通常依赖人类判断(如专家集体判断)或基于论文元数据训练的机器学习模型。两者虽各有价值,但均存在局限:人类判断易受认知偏差和文献覆盖范围狭窄影响,而自动化评估难以捕捉上下文线索和可信度的细微信号。本文提出一种混合方法:构建一个预测市场,其中算法代理与人类参与者共同交易,联合估计一项已发表科学发现在未来控制性复现实验中被证实的可能性。代理基于数百项先前复现实验的结果进行训练,人类参与者则通过实时交易贡献领域知识。我们通过多轮真实实验,在不同学科群体中评估该混合方法,并与纯人工和纯算法基线对比。结果显示,除少数例外情况外,混合市场表现匹配或超越纯算法市场,生成更准确、更可靠的复现预测。
原文摘要 · Abstract (English)
Determining whether published scientific findings can successfully be replicated is a long-standing challenge in the empirical sciences. Existing approaches for replicability assessment typically rely either on human judgment, i.e., creative assembly of human experts, or on machine learning models trained on paper content metadata. While both approaches have demonstrated value, each also has important limitations. Human forecasts can be influenced by cognitive biases and narrow exposure to the research literature, while automated assessments often struggle to capture contextual cues and subtle signals of credibility. In this paper, we examine a hybrid approach. Specifically, we introduce a hybrid prediction market in which algorithmic agents trade alongside human participants to jointly estimate the likelihood that a published scientific finding will be corroborated via the outcome of a controlled replication study. Agents are trained on outcomes from hundreds of prior replication studies while human participants contribute domain knowledge through real-time trading. We evaluate this hybrid approach through multiple live experiments involving participants from different academic disciplines and compare its performance to artificial-only and human-only baselines. Our results show that, except for a few cases, hybrid markets match or outperform artificial prediction markets, producing more accurate and reliable replication forecasts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。