用大模型检测代码架构决策违规，效果因决策类型而异
Evaluating Large Language Models for Detecting Architectural Decision Violations
- 多模型协同筛选并验证架构决策违规
- 显式可编码决策准确率高，隐式决策效果差
- 适合辅助代码相关决策审查，非替代专家
架构决策记录（ADRs）对维护软件架构质量至关重要，但许多决策违规因缺乏系统化文档和自动化检测机制而未被发现。大型语言模型（LLMs）为大规模自动化架构推理提供了新可能。我们通过分析109个GitHub仓库中的980份ADRs，评估了多模型流程下LLMs识别决策违规的能力。该流程由一个LLM初步筛查，三个额外LLM独立验证推理过程。评估涵盖一致性、准确率、精确率与召回率，并辅以专家评审。结果表明，对于明确且可从代码推断的决策，模型表现一致且准确；而对于依赖部署配置或组织知识的隐式或部署导向型决策，准确率显著下降。因此，LLMs可有效支持架构决策合规性验证，但尚无法替代针对非代码聚焦决策的人类专家判断。
原文摘要 · Abstract (English)
Architectural Decision Records (ADRs) play a central role in maintaining software architecture quality, yet many decision violations go unnoticed because projects lack both systematic documentation and automated detection mechanisms. Recent advances in Large Language Models (LLMs) open up new possibilities for automating architectural reasoning at scale. We investigated how effectively LLMs can identify decision violations in open-source systems by examining their agreement, accuracy, and inherent limitations. Our study analyzed 980 ADRs across 109 GitHub repositories using a multi-model pipeline in which one LLM primary screens potential decision violations, and three additional LLMs independently validate the reasoning. We assessed agreement, accuracy, precision, and recall, and complemented the quantitative findings with expert evaluation. The models achieved substantial agreement and strong accuracy for explicit, code-inferable decisions. Accuracy falls short for implicit or deployment-oriented decisions that depend on deployment configuration or organizational knowledge. Therefore, LLMs can meaningfully support validation of architectural decision compliance; however, they are not yet replacing human expertise for decisions not focused on code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。