arXiv:2511.05874cs.AI2025-11被引 3

对比6种推理型大模型在代码生成中的思考过程,发现复杂任务易缺步骤。

An Empirical Study of Reasoning Steps in Thinking Code LLMs

  • 通过控制思考步数和人工评估,分析模型推理质量
  • 难任务中不完整推理占比超70%,简单任务成功率超90%
  • 适合关注AI代码生成可靠性的开发者与研究者

思维型大语言模型(Thinking LLMs)在生成答案前会输出显式的中间推理链,可能提升代码生成的透明度、可解释性和准确性。然而,这些推理链的质量仍缺乏系统研究。我们对六种先进推理模型(DeepSeek-R1、OpenAI-o3-mini、Claude-3.7-Sonnet-Thinking、Gemini-2.0-Flash-Thinking、Gemini-2.5-Flash、Qwen-QwQ)在BigCodeBench的100个不同难度的代码生成任务上进行了全面实证研究。通过统计推理步数与表达冗余度,进行受控的步数预算调整,并开展21名参与者的人工评估,涵盖效率、逻辑正确性和完整性三个维度。结果显示,针对性增加步数可提升部分模型/任务的解决率,适度减少步数在标准任务中通常不影响结果,但在难题中常导致失败。系统分析构建了推理缺陷分类体系,发现不完整性是主要失败模式。任务复杂度显著影响推理质量:难题的不完整率远高于标准任务。稳定性分析表明,思维型模型在不同计算量下保持一致的逻辑结构,且能自我纠正先前错误。本研究为当前思维型大模型在软件工程中的优劣提供了新洞见。

原文摘要 · Abstract (English)

Thinking Large Language Models (LLMs) generate explicit intermediate reasoning traces before final answers, potentially improving transparency, interpretability, and solution accuracy for code generation. However, the quality of these reasoning chains remains underexplored. We present a comprehensive empirical study examining the reasoning process and quality of thinking LLMs for code generation. We evaluate six state-of-the-art reasoning LLMs (DeepSeek-R1, OpenAI-o3-mini, Claude-3.7-Sonnet-Thinking, Gemini-2.0-Flash-Thinking, Gemini-2.5-Flash, and Qwen-QwQ) across 100 code generation tasks of varying difficulty from BigCodeBench. We quantify reasoning-chain structure through step counts and verbosity, conduct controlled step-budget adjustments, and perform a 21-participant human evaluation across three dimensions: efficiency, logical correctness, and completeness. Our step-count interventions reveal that targeted step increases can improve resolution rates for certain models/tasks, while modest reductions often preserve success on standard tasks, rarely on hard ones. Through systematic analysis, we develop a reasoning-problematic taxonomy, identifying completeness as the dominant failure mode. Task complexity significantly impacts reasoning quality; hard problems are substantially more prone to incompleteness than standard tasks. Our stability analysis demonstrates that thinking LLMs maintain consistent logical structures across computational effort levels and can self-correct previous errors. This study provides new insights into the strengths and limitations of current thinking LLMs in software engineering.

代码生成推理链大模型评测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。