arXiv:2607.12463cs.AIcs.CL2026-07被引 1

让编程代理学会在函数调用中推理,提升代码生成与工具使用能力。

Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models

  • 基于函数调用结构设计填空训练,利用程序依赖图筛选复杂可推断的函数
  • 7B/14B/Qwen3-8B模型在SWE-Bench上验证率提升2.8~3.2个百分点
  • 训练后仍能改善非代码工具使用任务,适合构建高效编程代理

编程代理需整合外部工具返回结果以持续推理,但标准左到右预训练仅支持正向传播。我们发现编程代理的动作-观察-继续循环在结构上等同于函数调用:调用方绑定参数,被调方返回计算结果,下游代码消费该值。此类结构在互联网规模代码中普遍存在。我们提出函数感知填空训练(FIM)中段训练方法,通过程序依赖图分析和复杂性-可推断性双准则选取掩码函数。在26亿词元去污染语料(来自968个GitHub仓库)上对Qwen2.5-Coder-Instruct(7B/14B)及Qwen3-8B进行中段训练,再应用现有代理后训练流程。中段训练使SWE-Bench-Verified得分提升+2.8/+3.0(7B/14B)和+3.2(Qwen3-8B);SWE-Bench-Lite得分提升+3.7/+4.0/+5.4。效果在两种后训练管道(R2E-Gym、SWE-Smith)及非Qwen2.5基座(Qwen3-8B + SWE-Lego)上均成立。尽管训练语料仅含Python代码,函数调用归纳偏置仍保留在后训练中,并带来一致性能提升。

原文摘要 · Abstract (English)

Coding agents must integrate external tool returns into ongoing reasoning - a capability that standard left-to-right pretraining on code exposes only in its forward direction. We observe that the action-observation-continuation loop of a coding agent is structurally isomorphic to a function call site, where a caller binds arguments, a callee returns a value computed elsewhere, and downstream code consumes that value. This conditioning structure exists at internet scale in ordinary code. We exploit it through function-aware fill-in-the-middle (FIM) mid-training: a self-supervised objective that masks functions selected via program dependency graph analysis and a complexity-inferability double criterion. We mid-train Qwen2.5-Coder-Instruct (7B/14B) and Qwen3-8B on a 2.6B-token decontaminated corpus drawn from 968 GitHub repositories, then apply existing agentic post-training pipelines. Mid-training improves SWE-Bench-Verified by +2.8/+3.0 at 7B/14B and by +3.2 on Qwen3-8B; SWE-Bench-Lite gains are +3.7/+4.0/+5.4 on the same models. The improvement holds across two post-training pipelines (R2E-Gym, SWE-Smith) and on a non-Qwen2.5 base (Qwen3-8B with SWE-Lego). Beyond in-domain gains, mid-training also mitigates the capability erosion that agentic post-training otherwise inflicts on non-agent coding (e.g., LiveCodeBench) and non-coding tool-use benchmarks (tau-bench, BFCL): although the mid-training corpus contains Python code only, the function-call inductive bias survives post-training and yields consistent gains.

编程代理函数调用中段训练代码生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。