评测大模型写代码时对开发者要求的遵守程度，发现多数模型只部分满足约束。
CIFE: Code Instruction-Following Evaluation
- 构建1000个带7类约束的Python任务，用人类与AI协作确保约束清晰可测。
- 14个模型平均仅39%-66%严格遵守约束，即使正确率高也常忽略格式安全等要求。
- 提出C2A评分融合正确性与约束符合度，适合关注代码可靠性的开发者参考。
大型语言模型在真实代码生成中应用日益广泛，但仅功能正确不足以保证部署可靠性，开发者还期望模型遵循显式要求以确保鲁棒性、格式规范和安全性。现有基准主要通过测试用例执行评估正确性，难以反映模型对各类约束的实际遵守情况。本文引入一个包含1000个Python任务的基准，每个任务平均配有7个开发者指定的约束，覆盖13个类别。约束通过四阶段人机协作流程筛选，确保其原子性、相关性和客观性。我们采用互补的合规度量指标评估14个开源与闭源模型，并提出C2A分数，综合衡量正确性与约束符合程度。结果显示，部分满足与严格满足之间存在显著差距；表现优异的模型虽能达到超过90%的部分遵守率，但严格遵守率仍仅为39%-66%。这些发现表明，可信的代码生成不仅需要正确性，还需一致遵循开发者意图。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly applied to real-world code generation, where functional correctness alone is insufficient for reliable deployment, developers also expect adherence to explicit requirements for robustness, formatting, and security. Existing benchmarks primarily assess correctness through test-case execution, offering limited insight into how reliably models follow such constraints. We introduce a benchmark of 1,000 Python tasks, each paired with an average of 7 developer-specified constraints spanning 13 categories. Constraints are curated through a four-stage human-LLM pipeline to ensure they are atomic, relevant, and objective. We evaluate 14 open- and closed-source models using complementary adherence metrics and propose the C2A Score, a composite measure that jointly captures correctness and constraint compliance. Results reveal a substantial gap between partial and strict satisfaction, while strong models achieve over 90% partial adherence, strict adherence remains between 39-66%. These findings highlight that trustworthy code generation requires not only correctness but also consistent adherence to developer intent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。