指令微调后代码模型无需后处理即可生成连贯代码，但随机片段仍需裁剪。
From Output to Evaluation: Does Raw Instruction-Tuned Code LLMs Output Suffice for Fill-in-the-Middle Code Generation?
- 对比微调前后输出，发现指令微调显著提升代码上下文融合能力。
- 在完整行作为中间段时，微调模型无需后处理即达最佳性能。
- 随机代码片段仍需裁剪，因模型对输出边界缺乏感知。
在填空式代码生成任务中，原始输出常含冗余代码，影响自动评估效果。本文研究指令微调后的大型语言模型是否仍需后处理。实验表明，监督微调显著提升模型生成质量，使微调后的Qwen2.5-Coder（base与instruct版）在HumanEval Infilling与SAFIM基准测试中无需后处理即可取得更优表现，尤其当中间部分为完整代码行时。然而，当中间部分为随机代码片段时，仍需通过截断等后处理手段才能有效评估。
原文摘要 · Abstract (English)
Post-processing is crucial for the automatic evaluation of LLMs in fill-in-the-middle (FIM) code generation due to the frequent presence of extraneous code in raw outputs. This extraneous generation suggests a lack of awareness regarding output boundaries, requiring truncation for effective evaluation. The determination of an optimal truncation strategy, however, often proves intricate, particularly when the scope includes several programming languages. This study investigates the necessity of post-processing instruction-tuned LLM outputs. Our findings reveal that supervised fine-tuning significantly enhances FIM code generation, enabling LLMs to generate code that seamlessly integrates with the surrounding context. Evaluating our fine-tuned \texttt{Qwen2.5-Coder} (base and instruct) models on HumanEval Infilling and SAFIM benchmarks demonstrates improved performances without post-processing, especially when the \emph{middle} consist of complete lines. However, post-processing of the LLM outputs remains necessary when the \emph{middle} is a random span of code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。