用语法树结构掩码提升代码模型的补全能力
Structure-Aware Fill-in-the-Middle Pretraining for Code
- 基于抽象语法树掩码代码结构，而非随机字符
- 在12种语言上实测，性能比传统方法高5个百分点
- 适合需要精准代码编辑的开发者和研究者
填空式预训练（FIM）是代码大模型常用方法，但现有模型将代码视为纯文本，随机掩码字符片段。本文提出AST-FIM，利用抽象语法树（AST）在大规模上掩码完整的语法结构，如代码块、表达式或函数，使训练样本更契合通用代码结构与常见编辑模式。为评估真实场景下的填空任务，我们构建了基于3万+个GitHub提交的Real-FIM-Eval基准，涵盖12种语言。在10亿和80亿参数模型上的实验表明，AST-FIM在标准FIM基准上相比传统随机字符掩码方法性能提升最高达5个百分点，尤其适用于真实代码编辑任务。代码已开源：https://github.com/gonglinyuan/ast_fim。
原文摘要 · Abstract (English)
Fill-in-the-Middle (FIM) is a common pretraining method for code LLMs, where models complete code segments given surrounding context. However, existing LLMs treat code as plain text and mask random character spans. We propose and evaluate AST-FIM, a pretraining strategy that leverages Abstract Syntax Trees (ASTs) to mask complete syntactic structures at scale, ensuring coherent training examples better aligned with universal code structures and common code editing patterns such as blocks, expressions, or functions. To evaluate real-world fill-in-the-middle (FIM) programming tasks, we introduce Real-FIM-Eval, a benchmark derived from 30,000+ GitHub commits across 12 languages. On infilling tasks, experiments on 1B and 8B parameter models show that AST-FIM is particularly beneficial for real-world code editing as it outperforms standard random-character FIM by up to 5 pts on standard FIM benchmarks. Our code is publicly available at https://github.com/gonglinyuan/ast_fim.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。