通过课程学习提升小模型代码补全效果,兼顾准确率与实时性。
Improving FIM Code Completions via Context & Curriculum Based Learning
- 用难例构建课程数据集,结合上下文训练提升补全能力。
- 小模型在多线补全任务上性能显著提升,优于大模型但延迟更低。
- 适合对响应速度敏感的实时编程辅助场景。
填空式中间(FIM)模型在代码补全中至关重要,利用前缀和后缀上下文提供更准确、上下文相关的建议。本文提出通过引入上下文与课程学习样例来改进FIM代码补全,同时应对实时编码辅助中的低延迟挑战。我们识别出补全失败频繁的模式,揭示了小语言模型难以处理的复杂性。为此,从代码仓库中提取难补全模式构建课程数据集,并使用语义与静态分析工具(如TSC编译器)生成上下文示例。在该增强数据集上微调不同规模的模型(包括StarCoder和DeepSeek)。评估涵盖三个维度:Santa Coder FIM任务、Amazon CCEval基准以及基于SWE-bench的新多行填入评估基准。跨多种模型规模的全面消融实验表明,所有微调模型均有提升,尤其小模型在引入困难样本后性能增长更显著。这一发现对低延迟代码补全任务尤为重要。尽管大型模型(如GPT、Claude)在多行补全中表现优异,但因高延迟难以应用;而我们的微调模型实现了性能与延迟的平衡。最后通过在线A/B测试验证,完成接受率(CAR)和持续使用率(CPR)均显著提升,且无延迟增加。
原文摘要 · Abstract (English)
Fill-in-the-Middle (FIM) models play a vital role in code completion tasks, leveraging both prefix and suffix context to provide more accurate and contextually relevant suggestions. This paper presents approaches to improve FIM code completion while addressing the challenge of maintaining low latency for real-time coding assistance. We enhance FIM code completion by incorporating context and curriculum examples in the training process. We identify patterns where completion suggestions fail more frequently, revealing complexities that smaller language models struggle with. To address these challenges, we develop a curriculum dataset by extracting hard-to-complete patterns from code repositories and generate context examples using semantic and static analysis tools (e.g. TSC compiler). We fine-tune various sized models, including StarCoder and DeepSeek, on this enhanced dataset. Our evaluation encompasses three key dimensions: the Santa Coder FIM task, the Amazon CCEval benchmark, and a new Multi-Line Infilling evaluation benchmark derived from SWE-bench. Comprehensive ablation studies across multiple model sizes reveal that while all fine-tuned models show improvements, the performance gains are more pronounced for smaller parameter models and incorporating difficult-to-complete examples, as part of curriculum learning, improves the code completion performance. This finding is particularly significant given the latency constraints of code completion tasks. While larger models like GPT and Claude perform well in multi-line completions but are prohibitively challenging to use given high latency, and our fine-tuned models achieve a balance between performance and latency. Finally, we validate our approach through online A/B testing, demonstrating tangible improvements in Completion Acceptance Rate (CAR) and Completion Persistence Rate (CPR), with zero latency impact.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。