LLM代码代理在复杂架构约束下表现急剧下降，暴露了其生成生产级后端代码的脆弱性。
Constraint Decay: The Fragility of LLM Agents in Backend Code Generation
- 通过统一API合约，分离结构复杂度对代码生成的影响。
- 结构约束越多，测试通过率平均下降30分，部分模型接近零分。
- 在依赖约定的框架中表现更差，数据层错误是主要问题。
大型语言模型（LLM）代理在宽松规范下表现出色，但在生产级软件中需严格遵守架构模式、数据库和对象关系映射等结构约束。现有基准常忽略这些非功能性需求，奖励功能正确但结构随意的方案。本文系统评估代理在多文件后端生成中处理结构约束的能力。通过在80个新项目任务和20个功能实现任务中固定统一API合约，并覆盖八个主流Web框架，采用端到端行为测试与静态验证双重评估，隔离结构复杂度影响。结果揭示‘约束衰减’现象：随着结构要求增加，代理性能显著下降。能力强的配置在从基线到完全指定任务中，断言通过率平均降低30分；部分弱配置接近零分。框架敏感性分析显示，代理在最小化显式框架（如Flask）中表现良好，但在高度依赖约定的环境（如FastAPI、Django）中平均表现较差。错误分析指出，数据层缺陷（如错误查询构造和ORM运行时违规）是主要原因。本研究强调，同时满足功能与结构要求仍是代码代理的关键挑战。
原文摘要 · Abstract (English)
Large Language Model (LLM) agents demonstrate strong performance in autonomous code generation under loose specifications. However, production-grade software requires strict adherence to structural constraints, such as architectural patterns, databases, and object-relational mappings. Existing benchmarks often overlook these non-functional requirements, rewarding functionally correct but structurally arbitrary solutions. We present a systematic study evaluating how well agents handle structural constraints in multi-file backend generation. By fixing a unified API contract across 80 greenfield generation tasks and 20 feature-implementation tasks spanning eight web frameworks, we isolate the effect of structural complexity using a dual evaluation with end-to-end behavioral tests and static verifiers. Our findings reveal a phenomenon of constraint decay: as structural requirements accumulate, agent performance exhibits a substantial decline. Capable configurations lose 30 points on average in assertion pass rates from baseline to fully specified tasks, while some weaker configurations approach zero. Framework sensitivity analysis exposes significant performance disparities: agents succeed in minimal, explicit frameworks (e.g., Flask) but perform substantially worse on average in convention-heavy environments (e.g., FastAPI, Django). Finally, error analysis identifies data-layer defects (e.g., incorrect query composition and ORM runtime violations) as the leading root causes. This work highlights that jointly satisfying functional and structural requirements remains a key open challenge for coding agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。