构建交互式反馈基准，提升大模型生成科研代码能力
RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedback
- 设计多轮交互式反馈机制，模拟真实科研协作流程
- 102个任务测试显示，丰富反馈使主流模型性能显著提升
- 适合研究智能编程、人机协同开发的学者参考
大型语言模型在支持科研代码实现方面展现出潜力，但其生成正确可执行代码的能力仍有限。现有工作大多采用单次生成设定，忽略了科研开发中迭代与反馈驱动的真实流程。为弥补这一差距，我们提出RECODE-H，一个包含102个来自论文和仓库的任务基准，通过多轮与大模型模拟的人类反馈交互来评估LLM代理。该基准包含结构化指令、单元测试及五级反馈层级，反映真实的研究员-代理协作模式。我们进一步提出ReCodeAgent框架，将反馈融入迭代式代码生成过程。对GPT-5、Claude-Sonnet-4、DeepSeek-V3.1和Gemini 2.5等主流LLM的实验表明，更丰富的反馈带来显著性能提升，同时揭示了生成复杂科研代码仍存在的挑战。RECODE-H为发展自适应、反馈驱动的LLM代理在科研实现中的应用奠定了基础。
原文摘要 · Abstract (English)
Large language models (LLMs) show the promise in supporting scientific research implementation, yet their ability to generate correct and executable code remains limited. Existing works largely adopt one-shot settings, ignoring the iterative and feedback-driven nature of realistic workflows of scientific research development. To address this gap, we present RECODE-H, a benchmark of 102 tasks from research papers and repositories that evaluates LLM agents through multi-turn interactions with LLM-simulated human feedback. It includes structured instructions,unit tests, and a five-level feedback hierarchy to reflect realistic researcher-agent collaboration. We further present ReCodeAgent, a framework that integrates feedback into iterative code generation. Experiments with leading LLMs, including GPT-5, Claude-Sonnet-4, DeepSeek-V3.1, and Gemini 2.5, show substantial performance gains with richer feedback, while also highlighting ongoing challenges in the generation of complex research code. RECODE-H establishes a foundation for developing adaptive, feedback-driven LLM agents in scientific research implementation
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。