指令微调让代码模型更懂指令，但会削弱自动补全能力。
Lost in the Flow with Code Talkers: Unveiling the Instruction-Tuning Tax of Large Language Models in Code Tasks
- 发现指令微调带来性能权衡：懂指令但补全变差。
- 实验表明指令模型在补全任务上表现下降15%以上。
- 适合需要理解自然语言指令的场景，不推荐纯补全任务。
AI代码助手显著提升了开发者效率，通过自动生成符合意图的代码，已广泛集成至集成开发环境（IDE）。开发者在编程中存在两种认知模式：流程模式（Flow）与指令模式（Command）。流程模式下需工具完成未完成代码的自动补全，而指令模式下则需将自然语言指令转化为可执行代码。尽管指令微调的大语言模型（LLMs）在多数场景中表现优异，但其对不同代码任务的适用性仍不明确。为此，本文首次实证研究揭示了指令微调在编程模式间的隐性代价，称之为「指令微调税」。结果表明：指令微调并非免费红利——虽增强了指令遵循与结构化引导理解能力，但往往以牺牲代码补全性能为代价。通过定性与定量分析，包括人工错误分类、生成保真度指标及训练过程中的中间检查点评估，我们总结出七项发现与四项启示。本研究为构建更高效的智能编程工具提供了新视角，强调需在指令遵循与代码生成能力间谨慎权衡。
原文摘要 · Abstract (English)
AI coding assistants have significantly improved developer productivity by automatically suggesting code that aligns with user intent, and many of these tools are now integrated directly into Integrated Development Environments (IDEs). Developers interact with code in two distinct cognitive modes: Flow and Command. While developers require tools that directly complete or infill code in unfinished programs during Flow mode, they also need tools that can comprehend intentions expressed as natural-language instructions and convert them into executable code in Command mode. Although instruction-tuned Large Language Models (LLMs) dominate many application scenarios due to their abilities to infer and fulfill developers' intents, it remains unclear whether the same paradigm is equally suitable for different code-related tasks. Therefore, it is necessary to understand how instruction tuning affects the feasibility of CodeLLMs as coding assistants. To fill this gap, we conduct the first empirical study that uncovers a key trade-off caused by instruction tuning across programming modes, which we term the Instruction-Tuning Tax. Our results show that instruction tuning is not a free lunch: although instruction-tuned models are more capable of following instructions and leveraging structured guidance, these gains often come at the cost of weaker infilling performance. We further extend our study through both qualitative and quantitative analyses, including manual failure categorization, behavioral metrics that capture generation fidelity, and intermediate-checkpoint evaluation throughout the tuning process. Summarizing our results into seven findings and four implications, our study offers a new perspective on the development of AI-powered coding tools and highlights the need to carefully balance instruction-following ability with effective code generation assistance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。