让代码评估同时考量功能与用户偏好,提升AI生成代码的自然度。
SWE-IF: Aligning Code Evaluation with Human Preference
- 构建30条可验证代码指令并设计自动检查器
- 强模型仍难同时满足多条非功能性指令,且功能会退化
- 指令遵循能力是区分模型好坏的关键,更贴近人类偏好
大型语言模型(LLMs)推动了“ vibe coding”模式,用户通过自然语言交互生成并迭代优化代码,直到其“感觉对”。这种“感觉对”不仅要求功能正确,还强调代码可读性、意图保留和表达自然。然而现有评估仍依赖pass@k指标,仅关注功能正确性,忽略用户常提的非功能性要求。本文提出:指令遵循是vibe check的核心缺失环节。为此,我们构建VeriCode——一套包含30条可验证代码指令的分类体系及确定性验证器,并将其融入主流评测基准,形成SWE-IF测试平台,用于同时评估功能正确性与指令遵循能力。在31个LLM上的评估显示,即使最强模型也难以同时满足多项指令,且存在功能退化现象。更重要的是,综合功能正确性与指令遵循的评分与人类偏好相关性最高,其中指令遵循成为模型间的主要区分因素。代码、数据与分类体系已开源。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural language interactions until it passes their vibe check. Vibe check reflects human preference and goes beyond functionality: the solution should feel right, read cleanly, preserve intent, and remain correct. However, current code evaluation remains anchored to pass@k and captures only functional correctness, overlooking non-functional instructions that users routinely apply. In this paper, we hypothesize that instruction following is the missing piece underlying vibe check besides functional correctness. To quantify models' code instruction-following capabilities with measurable signals, we present VeriCode, a taxonomy of 30 verifiable code instructions together with deterministic verifiers. We use the taxonomy to augment established evaluation suites, resulting in SWE-IF, a testbed to assess both instruction following and functional correctness. Evaluating 31 LLMs, we show that even the strongest models struggle to comply with multiple instructions and exhibit functional regression. Most importantly, a composite score of functional correctness and instruction following correlates best with human preference, with instruction following emerging as the primary differentiator among LLMs. Our code, data, and taxonomy are available at https://github.com/maszhongming/SWE-IF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。