提升代码模型按指令生成代码的能力，更贴合实际开发要求。
IFEvalCode: Controlled Code Generation
- 提出正向与反向约束生成方法，增强模型遵循指令能力。
- 在40多个模型上测试，闭源模型表现优于开源模型。
- 新基准支持中英文双语评估，区分正确性与指令遵循度。
代码大语言模型在将自然语言描述转化为功能性代码方面取得了显著进展；然而，真实应用场景往往需要严格遵守编码风格、行数限制和结构约束等详细要求，远超基本正确性。为此，本文提出正向与反向约束生成方法，以提升代码大模型在可控代码生成中的指令遵循能力，使输出更贴近人为定义的规范。作者进一步构建了IFEvalCode多语言基准，涵盖7种编程语言（Python、Java、JavaScript、TypeScript、Shell、C++、C#），共1600个测试样本，每个样本均配有中英文查询。该基准创新性地将评估解耦为两个指标：正确性（Corr.）与指令遵循度（Instr.），实现更细致的性能分析。对40余种LLM的实验表明，闭源模型在可控代码生成中优于开源模型，并揭示了模型生成正确代码与精准遵循指令之间存在显著差距。
原文摘要 · Abstract (English)
Code large language models (Code LLMs) have made significant progress in code generation by translating natural language descriptions into functional code; however, real-world applications often demand stricter adherence to detailed requirements such as coding style, line count, and structural constraints, beyond mere correctness. To address this, the paper introduces forward and backward constraints generation to improve the instruction-following capabilities of Code LLMs in controlled code generation, ensuring outputs align more closely with human-defined guidelines. The authors further present IFEvalCode, a multilingual benchmark comprising 1.6K test samples across seven programming languages (Python, Java, JavaScript, TypeScript, Shell, C++, and C#), with each sample featuring both Chinese and English queries. Unlike existing benchmarks, IFEvalCode decouples evaluation into two metrics: correctness (Corr.) and instruction-following (Instr.), enabling a more nuanced assessment. Experiments on over 40 LLMs reveal that closed-source models outperform open-source ones in controllable code generation and highlight a significant gap between the models' ability to generate correct code versus code that precisely follows instructions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。