首次系统评估代码生成对运行环境的适应能力，发现现有大模型表现不佳。
Environment-Aware Code Generation: How far are We?
- 构建多包、可执行验证、关注弃用的基准数据集VersiBCB
- 三类适配策略使代码在复杂环境中可执行率提升23.6%
- 适合关注实际工程落地的大模型研究者
大语言模型在代码生成方面取得进展，但多数评测仍局限于默认或未指定软件环境下的小规模代码（如单个函数）。这使得我们难以判断大模型能否为用户特定环境生成可直接运行的代码。本文首次系统研究环境感知代码生成（EACG），要求生成的代码在任意软件配置下功能正确且可执行。为此，我们提出VersiBCB基准，具备多包、执行验证和弃用感知特性，捕捉以往数据集常忽略的复杂演化环境。基于该基准，我们探究数据、参数和缓存三类适应机制，设计代表性策略。结果表明，当前大模型在环境相关代码生成上表现较差，而我们的适配方法显著提升了环境兼容性与可执行性。研究揭示了大模型在实际软件工程中部署的关键挑战与机遇。
原文摘要 · Abstract (English)
Recent progress in large language models (LLMs) has improved code generation, but most evaluations still test isolated, small-scale code (e.g., a single function) under default or unspecified software environments. As a result, it is unclear whether LLMs can reliably generate executable code tailored to a user's specific environment. We present the first systematic study of Environment-Aware Code Generation (EACG), where generated code must be functionally correct and directly executable under arbitrary software configurations. To enable realistic evaluation, we introduce VersiBCB, a benchmark that is multi-package, execution-verified, and deprecation-aware, capturing complex and evolving environments that prior datasets often overlook. Using VersiBCB, we investigate three complementary adaptation axes: data, parameters, and cache, and develop representative strategies for each. Our results show that current LLMs struggle with environment-specific code generation, while our adaptations improve environment compatibility and executability. These findings highlight key challenges and opportunities for deploying LLMs in practical software engineering workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。