用更少数据提升代码补全模型，仅10亿token就达顶尖性能
On Pretraining for Project-Level Code Completion
- 通过优化仓库处理策略扩展上下文至16384词元
- 仅用10亿令牌训练即在长代码评测中媲美大模型
- 新位置编码适配让小数据也能高效训练
仓库级预训练常用于使大语言模型利用代码库全局上下文，从而生成更准确的代码补全。本文研究不同仓库处理策略对OpenCoder（15亿参数）上下文学习的影响。通过在额外10亿个经过筛选的仓库级数据上训练，将上下文窗口从4096扩展至16384词元。尽管使用数据量远小于同类模型（通常数百亿词元），该模型在Long Code Arena基准测试中仍取得相当表现。我们发现多种仓库处理技术效果相近，主要提升来自适应新的旋转位置编码（RoPE）缩放参数。最后，我们表明在原始序列长度下采用更简单的文件级训练依然高效，为资源受限场景下的仓库级代码补全研究提供了可行路径。
原文摘要 · Abstract (English)
Repository-level pretraining is commonly used to enable large language models for code to leverage codebase-wide context. This enhances their ability to generate accurate and context-aware code completions. In this work, we investigate how different repository-processing strategies affect in-context learning in OpenCoder, a 1.5B-parameter model. We extend its context window from 4,096 to 16,384 tokens by training on additional 1B tokens of curated repository-level data. Despite relying on a smaller dataset than competing models (which often use hundreds of billions of tokens), our model achieves comparable performance on the Long Code Arena benchmark. We find that various repository-processing techniques yield similarly strong results, with the primary gain coming from adapting to a new rotary positional embedding (RoPE) scaling parameter. Finally, we show that a simpler file-level training approach at the original sequence length remains highly effective, opening up repository-level code completion research to settings with more constrained data and compute resources.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。