强编码代理靠元编程应对陌生语言,弱代理则难以模仿。
Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languages
- 用元编程生成目标语言代码,而非直接编写
- 禁用元编程后性能大幅下降,证明其关键作用
- 适合研究智能体适应性与工具使用机制的学者
基于大模型的编码代理通常在主流语言和公开仓库中评估,但这类基准难以反映其面对陌生编程语言时的表现。本文在四种冷门语言上,采用文件编辑、本地执行和隐藏测试评分的连续流程,评估六种主流编码代理。结果发现,表现最强的Claude Opus 4.6与GPT-5.4 xhigh常避免直接编写目标语言代码,转而用Python生成目标代码并本地调试。禁止此元编程策略导致性能显著下降。从该策略中提炼的文本指导对弱代理改善有限。相比之下,仅提供由Opus生成的辅助Python代码(无解题程序或答案),即显著提升Sonnet 4.6和GPT-5.4 mini的表现,而Haiku 4.5仍低。更多解释器调用与输出 token 提升强代理表现,但对弱代理影响微小,表明资源只放大已有策略而非创造新能力。整体表明,强代理通过工具、反馈与工作区状态构建对目标语言规则的理解,元编程是最明显例证,核心差异在于能否构建并调试有效策略。
原文摘要 · Abstract (English)
LLM-based coding agents are usually evaluated in familiar software settings: mainstream languages, common libraries, and public repositories. These benchmarks remain important, but they can hide how agents behave when the language itself is unfamiliar. We evaluate six contemporary coding agents on four esoteric programming languages using a sequential setup with file editing, local execution, and hidden-test grading. Our protocol exposes capability differences between these agents that mainstream coding and agentic benchmarks such as SWE-Bench Verified and Terminal-Bench 2.0 compress into much narrower bands. We observe that the strongest agents, Claude Opus 4.6 and GPT-5.4 xhigh, often avoid writing the target language directly. On Brainfuck and Befunge-98, they write Python programs that generate target-language code and debug those generators locally. Forbidding this metaprogramming strategy causes large performance drops. Text guidance distilled from this strategy does not materially improve weaker agents. In contrast, Opus-derived Python helper code for building generators, with no solved benchmark programs or hidden-test answers, sharply improves Sonnet 4.6 and GPT-5.4 mini on the same problems, while Haiku 4.5 remains low. More interpreter calls and output tokens improve stronger agents but leave weaker agents near their original performance, indicating that these resources amplify useful strategies rather than create them. Together, these results show that strong coding agents adapt to unfamiliar languages by using tools, feedback, and workspace state to build a working model of the target language. Metaprogramming is the clearest case, but the broader gap is constructing and debugging a strategy that works under the target language's rules.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。