首个评测代码生成中指令遵循能力的基准，助力提升AI编程可靠性。
CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generation
- 构建多任务代码生成评估框架，涵盖函数生成、调试、重构等场景。
- 实测显示主流模型在复杂指令下准确率不足60%，存在显著偏差。
- 适合研究AI编程、模型对齐与自动化开发工具的开发者使用。
随着大语言模型（LLMs）的快速发展，代码生成任务中对强指令遵循能力的需求日益增长。代码生成不仅加速原型设计与自动化测试，还能通过提升代码可维护性与复用性增强开发者效率。本文提出CodeIF，首个专门用于评估LLMs在多样化代码生成场景中遵循任务导向指令能力的基准。CodeIF覆盖函数合成、错误调试、算法重构和代码解释等多种任务，提供跨复杂度与编程领域全面评估。我们对多种LLMs进行大规模实验，分析其在满足任务需求上的优势与局限。结果揭示当前模型在指令对齐方面的表现，以及生成一致性、可维护性与上下文相关代码的能力。研究强调了指令遵循型LLMs在现代软件开发中的关键作用，并为未来提升模型适应性、可靠性和整体效能指明方向。CodeIF数据与代码已公开：https://github.com/lin-rany/codeIF
原文摘要 · Abstract (English)
With the rapid advancement of Large Language Models (LLMs), the demand for robust instruction-following capabilities in code generation tasks has grown significantly. Code generation not only facilitates faster prototyping and automated testing, but also augments developer efficiency through improved maintainability and reusability of code. In this paper, we introduce CodeIF, the first benchmark specifically designed to assess the abilities of LLMs to adhere to task-oriented instructions within diverse code generation scenarios. CodeIF encompasses a broad range of tasks, including function synthesis, error debugging, algorithmic refactoring, and code explanation, thereby providing a comprehensive suite to evaluate model performance across varying complexity levels and programming domains. We conduct extensive experiments with LLMs, analyzing their strengths and limitations in meeting the demands of these tasks. The experimental results offer valuable insights into how well current models align with human instructions, as well as the extent to which they can generate consistent, maintainable, and contextually relevant code. Our findings not only underscore the critical role that instruction-following LLMs can play in modern software development, but also illuminate pathways for future research aimed at enhancing their adaptability, reliability, and overall effectiveness in automated code generation. CodeIF data and code are publicly available: https://github.com/lin-rany/codeIF
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。