通过反事实数据提升代码大模型对细节的敏感度，显著改善性能。
Success is in the Details: Evaluate and Enhance Details Sensitivity of Code LLMs through Counterfactuals
- 用反事实扰动构建新基准CTF-Code，微小输入变化引发显著输出差异。
- 多模型在原题上表现下降超10%，暴露对细节不敏感问题。
- 引入CTF-Instruct微调框架，提升模型在多个数据集上的准确率。
代码敏感性指代码大模型识别并响应问题描述中细微变化的能力。现有代码评测基准和指令数据主要关注难度与多样性，忽视了敏感性。本文提出CTF-Code基准，通过反事实扰动构造，使输入变化极小而输出变化最大化。评估显示，多个大模型在该基准上性能下降超过10%。为提升敏感性，本文设计CTF-Instruct微调框架，在现有数据基础上增加反事实样本，并采用选择机制兼顾难度、多样性和敏感性三个维度。实验表明，使用CTF-Instruct微调后的模型在CTF-Code上提升超2%，在LiveCodeBench上性能提升超过10%，验证了增强敏感性可有效提升模型性能。
原文摘要 · Abstract (English)
Code Sensitivity refers to the ability of Code LLMs to recognize and respond to details changes in problem descriptions. While current code benchmarks and instruction data focus on difficulty and diversity, sensitivity is overlooked. We first introduce the CTF-Code benchmark, constructed using counterfactual perturbations, minimizing input changes while maximizing output changes. The evaluation shows that many LLMs have a more than 10\% performance drop compared to the original problems. To fully utilize sensitivity, CTF-Instruct, an incremental instruction fine-tuning framework, extends on existing data and uses a selection mechanism to meet the three dimensions of difficulty, diversity, and sensitivity. Experiments show that LLMs fine-tuned with CTF-Instruct data achieve over a 2\% improvement on CTF-Code, and more than a 10\% performance boost on LiveCodeBench, validating the feasibility of enhancing LLMs' sensitivity to improve performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。