让大模型在推理时学会陌生编程语言,无需大量训练数据。
Bridging the Knowledge Void: Inference-time Acquisition of Unfamiliar Programming Languages for Coding Tasks
- 通过与官方文档和执行环境互动,动态学习新语言
- 在代码生成等任务上超越检索增强基线模型
- 适合需要快速适配新语言的研究者和开发者
大型语言模型(LLMs)在编程任务中的表现通常依赖于其大规模预训练语料,但面对未见过的编程语言时性能会急剧下降。本文提出一种推理时语言获取(ILA)范式,让模型通过与有限外部资源的动态交互掌握陌生语言。我们构建了ILA-agent框架,赋予大模型一系列类人行为工具,使其能结构化地探索、应用和验证语言知识。为严格评估低资源场景下的效果,我们设计了基于新型静态类型语言Cangjie的多任务基准Cangjie-bench。在代码生成、翻译和程序修复任务中,ILA-agent显著优于检索增强基线。轨迹分析揭示了涌现的行为模式,也暴露了仍存在的性能差距。
原文摘要 · Abstract (English)
The proficiency of Large Language Models (LLMs) in coding tasks is often a reflection of their extensive pre-training corpora, which typically collapses when confronted with previously unfamiliar programming languages. Departing from data-intensive finetuning, we investigate the paradigm of Inference-time Language Acquisition (ILA), where an LLM masters an unfamiliar language through dynamic interaction with limited external resources. In this paper, we propose ILA-agent, a general ILA framework that equips LLMs with a set of behavioral primitives. By modeling essential human-like behaviors as a suite of tools, ILA-agent enables LLMs to incrementally explore, apply, and verify language knowledge through structured interactions with the official documentation and execution environment. To provide a rigorous evaluation in a low-resource setting, we construct Cangjie-bench, a multi-task benchmark based on the novel statically-typed language Cangjie. We instantiate ILA-agent for Cangjie and evaluate its performance across code generation, translation, and program repair tasks. Results using diverse LLMs demonstrate that ILA-agent significantly outperforms retrieval-augmented baselines. Further analysis of agent trajectories characterizes the emergent behavior patterns while highlighting persisting performance gaps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。