大模型生成代码看似正确,实则结构混乱,部署后常出问题。
The Patchwork Problem in LLM-Generated Code

- 用图结构分析代码各组件间的依赖关系,发现隐藏的结构性错误。
- 多数错误绕过类型检查、测试和静态分析,且不同模型表现差异大。
- 适合关注AI编程质量、安全与工程落地的研究者和开发者。
大模型生成的代码常能通过编译、通过测试并看似正确,但部署后却失效。根本原因通常是结构性而非逻辑性问题:如调用未声明的配置键、引用不存在的包、新路由遗漏认证保护等。每个片段局部有效,但整体不连贯,现有CI工具难以发现此类问题。本文将此现象称为“拼贴问题”,形式化了代码仓库中多种图结构(导入、调用、依赖、配置、模式、资源、控制流、路由)的一致性约束,并提出八类缺陷分类体系,区分专属于大模型生成的错误与仅被放大的错误。设计了一种混合验证框架,对成熟静态分析工具擅长领域复用,对跨领域一致性约束使用定制检测器,聚焦可证明的约束违反而非启发式匹配。在两个前沿模型上,四种提示策略下的实证评估显示,绝大多数结构性错误逃过了类型检查、测试和SAST;且不同模型的错误模式存在质的区别,挑战了通用缓解策略。在真实世界AI生成代码仓库上的外部验证确认,这些问题并非实验构造,而是无监督编程场景下普遍存在的风险。
原文摘要 · Abstract (English)
LLM-generated code often compiles, passes tests, and appears correct, yet breaks once deployed. The root cause is frequently structural rather than logical. A generated endpoint references configuration keys never declared in the project, an import targets a package that does not exist in any registry, or a new route omits the authentication guard applied to every sibling endpoint. Each patch is locally valid but globally incoherent, and standard CI toolchains rarely surface these failures. As LLM-powered coding tools see widespread adoption, this blind spot poses a growing risk to software quality. We call this the \textbf{patchwork problem}. This paper formalizes structural coherence as consistency invariants over graph representations of repository artifacts, including import, call, dependency, configuration, schema, resource, control-flow, and routing graphs, and introduces an eight-category failure taxonomy distinguishing defects specific to LLM generation from those merely amplified by it. We present a hybrid verification framework that delegates to mature static analysis tools where they already excel and deploys purpose-built detectors for cross-cutting invariants underserved by existing toolchains, targeting provable constraint violations rather than heuristic pattern matching. Empirical evaluation across two frontier models under four prompting strategies reveals that the vast majority of structural failures evade type checking, testing, and SAST entirely, and that failure patterns diverge qualitatively between models in ways that challenge model-agnostic mitigation strategies. External validation on real-world AI-generated repositories confirms that these failures are not artifacts of controlled experimentation but are prevalent wherever LLMs write code with minimal human oversight.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。