LLM生成代码常因依赖缺失无法运行，真实依赖远超声明数量。
AI-Generated Code Is Not Reproducible (Yet): An Empirical Study of Dependency Gaps in LLM-Based Coding Agents
- 构建三层依赖框架，区分声明、可用与运行时依赖
- 仅68.3%代码可直接运行，Python成功率89.2%，Java仅44.0%
- 实际运行依赖是声明值的13.5倍，存在严重隐藏依赖
大型语言模型（LLMs）作为编程代理有望加速软件开发，但其对生成代码可复现性的影响仍不明确。本文通过实证研究，评估三种前沿LLM编程代理（Claude Code、OpenAI Codex、Gemini）在Python、JavaScript和Java中，基于100个标准化提示生成的300个项目在仅安装操作系统包且仅使用模型声明依赖的纯净环境中是否能成功执行。我们引入三层依赖框架（区分声称、可用、运行时依赖），量化执行可复现性。结果表明，仅有68.3%的项目可即刻运行，语言间差异显著（Python 89.2%，Java 44.0%）。平均而言，从声明依赖到实际运行依赖扩大了13.5倍，暴露出大量隐藏依赖。
原文摘要 · Abstract (English)
The rise of Large Language Models (LLMs) as coding agents promises to accelerate software development, but their impact on generated code reproducibility remains largely unexplored. This paper presents an empirical study investigating whether LLM-generated code can be executed successfully in a clean environment with only OS packages and using only the dependencies that the model specifies. We evaluate three state-of-the-art LLM coding agents (Claude Code, OpenAI Codex, and Gemini) across 300 projects generated from 100 standardized prompts in Python, JavaScript, and Java. We introduce a three-layer dependency framework (distinguishing between claimed, working, and runtime dependencies) to quantify execution reproducibility. Our results show that only 68.3% of projects execute out-of-the-box, with substantial variation across languages (Python 89.2%, Java 44.0%). We also find a 13.5 times average expansion from declared to actual runtime dependencies, revealing significant hidden dependencies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。