首个评估代码任务推理质量的基准，解决现有评测不适用问题
Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks
- 构建涵盖生成、摘要、分类三类任务的代码推理评测基准
- 发现五类评测缺陷，提出两阶段验证模型VERA提升准确率
- 适用于代码生成与推理评估研究者，支持多数据集验证
大型语言模型（LLMs）在解决代码任务时越来越多依赖显式推理，但评估推理质量仍具挑战性。现有推理评估工具未针对代码设计，且当前基准主要聚焦代码生成，忽视其他编码任务。本文提出CodeRQ-Bench，首个用于评估代码任务中推理质量的基准，覆盖生成、摘要和分类三类任务。基于该基准，分析了1,069个现有评估器中的不匹配案例，识别出五类常见局限，并提炼出四项设计启示。据此提出VERA——一种结合证据驱动验证与模糊性感知评分修正的两阶段评估器。在CodeRQ-Bench上的实验表明，VERA在四个数据集上持续优于强基线，最高提升AUCROC达0.26，AUPRC达0.21。项目代码已开源，支持未来研究。
原文摘要 · Abstract (English)
Large language models (LLMs) increasingly rely on explicit reasoning to solve coding tasks, yet evaluating the quality of this reasoning remains challenging. Existing reasoning evaluators are not designed for coding, and current benchmarks focus primarily on code generation, leaving other coding tasks largely unexplored. We introduce CodeRQ-Bench, the first benchmark for evaluating LLM reasoning quality across three coding task categories: generation, summarization, and classification. Using this benchmark, we analyze 1,069 mismatch cases from existing evaluators, identify five recurring limitations, and derive four design insights for reasoning evaluation in coding tasks. Guided by these insights, we propose VERA, a two-stage evaluator that combines evidence-grounded verification with ambiguity-aware score correction. Experiments on CodeRQ-Bench show that VERA consistently outperforms strong baselines across four datasets, improving AUCROC by up to 0.26 and AUPRC by up to 0.21. We release CodeRQ-Bench at https://github.com/MrLYG/CodeRQ-Bench, supporting future investigations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。