让大模型更准判断研究创意新颖性，解决其总爱判成‘中等新颖’的偏见。
Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty
- 在推理过程中探测隐藏状态中的新颖性判断，动态调整最终结论。
- 在强基线基础上提升新颖性判断性能22.30%，显著缓解‘中等新颖’偏差。
- 适合需要精准评估科研创意新颖性的研究人员和AI评审系统。
自动化新颖性判断可加速科学发现，实现研究创意的高效评估、优化与比较。尽管大语言模型被广泛用于此任务，我们发现其判断能力存在此前未受重视的局限：尽管生成的推理过程与人类专家高度相似，但最终的新颖性判断却常出现显著偏差。我们证明，这种误校准源于系统性倾向将创意判定为“中等新颖”。为此，我们提出轻量级方法 Think-Probe-Respond（TPR），在推理阶段从隐藏状态中探测潜在的新颖性判断，并用该探测结果引导最终响应。在多个强基线上的实验表明，TPR使新颖性判断性能提升22.30%，并有效缓解了“中等新颖”这一普遍偏差。
原文摘要 · Abstract (English)
Automated novelty judgment can accelerate scientific discovery by enabling efficient evaluation, refinement, and comparison of research ideas. While large language models are increasingly adopted for this task, we investigate a previously overlooked limitation in their judgment capabilities: despite generating reasoning rationales that closely mirror those of human experts, their final novelty judgments often diverge substantially. We demonstrate that this miscalibration stems from a systematic bias towards judging ideas as "medium novel". To mitigate this, we propose Think-Probe-Respond (TPR), a lightweight approach that probes latent novelty judgments from hidden states during the reasoning phase and uses the probed judgments to condition the final response. Across strong baselines, TPR improves novelty judgment performance by 22.30% and successfully mitigates the prevalent "medium novelty" bias.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。