评测代码补全模型在快速迭代库中的表现并提出改进方法
LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation
- 构建版本感知的代码补全基准，涵盖8个热门库的演进过程
- 发现公共库更新显著降低模型性能，版本差异是主要挑战
- 通过引入版本化文档和提示工程提升模型对新版本库的适应性
近期代码补全模型多聚焦于局部文件上下文，但未能充分反映真实开发中依赖快速演进的公开库的复杂性。为此，我们提出 LibEvolutionEval，一个需理解库演进才能准确完成内联补全的任务基准。该基准包含8个库（torch、torchvision、scipy、pil、tqdm、pyyaml、matplotlib、pandas）在一年内的演化数据，并对两个主流维护库（PyTorch 和 Matplotlib）进行了详细分析。评估多个公开模型发现，库的版本演进显著影响模型表现。通过研究检索版本化库文档与提示策略的缓解效果，验证了其在应对快速演进包方面的有效性，为未来提升模型适应能力提供了可行路径。
原文摘要 · Abstract (English)
Recent advancements in code completion models have primarily focused on local file contexts. However, these studies do not fully capture the complexity of real-world software development, which often requires the use of rapidly-evolving public libraries. To fill the gap, we introduce LibEvolutionEval, a detailed study requiring an understanding of library evolution to perform in-line code completion accurately. LibEvolutionEval provides a version-specific code-completion task comprised of eight libraries (torch, torchvision, scipy, pil, tqdm, pyyaml, matplotlib, and pandas) as they evolve over the year along with a detailed analysis of the evolution of two popular and well-maintained public libraries: PyTorch and Matplotlib. We evaluate popular public models and find that public library evolution significantly influences model performance. We explored mitigation methods by studying how retrieved version-specific library documentation and prompting can improve the model's capability in handling these fast-evolving packages, paving a promising future path in better handling fast-evolving libraries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。