构建双语代码事实性评测集，提升大模型编程准确性。
CodeSimpleQA: Scaling Factuality in Code Large Language Models
- 设计双语代码事实性问答数据集，覆盖多语言与主流计算机领域。
- 通过6600万条指令微调+强化学习，显著提升模型代码事实性。
- 揭示顶尖模型仍存事实错误，强调事实对齐对可靠编程模型的重要性。
大型语言模型在代码生成方面取得显著进展，能根据自然语言指令生成代码片段。然而，确保模型对编程概念、技术实现等内容生成事实准确的回答仍是关键挑战。以往代码相关评测多关注代码执行正确性，忽视了编程知识的事实准确性。为此，我们提出CodeSimpleQA，一个全面的双语基准，用于评估代码大模型在回答代码相关问题时的事实准确性，包含精心筛选的中英文问答对，覆盖多种编程语言和主要计算机科学领域。此外，我们构建了包含6600万样本的CodeSimpleQA-Instruct大规模指令语料，并开发结合监督微调与强化学习的后训练框架。对多种大模型的综合评估显示，即使前沿模型在代码事实性方面仍有困难。所提框架相较基线模型有显著提升，凸显事实感知对齐在构建可靠代码大模型中的关键作用。
原文摘要 · Abstract (English)
Large language models (LLMs) have made significant strides in code generation, achieving impressive capabilities in synthesizing code snippets from natural language instructions. However, a critical challenge remains in ensuring LLMs generate factually accurate responses about programming concepts, technical implementations, etc. Most previous code-related benchmarks focus on code execution correctness, overlooking the factual accuracy of programming knowledge. To address this gap, we present CodeSimpleQA, a comprehensive bilingual benchmark designed to evaluate the factual accuracy of code LLMs in answering code-related questions, which contains carefully curated question-answer pairs in both English and Chinese, covering diverse programming languages and major computer science domains. Further, we create CodeSimpleQA-Instruct, a large-scale instruction corpus with 66M samples, and develop a post-training framework combining supervised fine-tuning and reinforcement learning. Our comprehensive evaluation of diverse LLMs reveals that even frontier LLMs struggle with code factuality. Our proposed framework demonstrates substantial improvements over the base model, underscoring the critical importance of factuality-aware alignment in developing reliable code LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。