用大模型检测微服务架构模式,跨语言识别效果参差不齐。
Can an LLM Detect Instances of Microservice Infrastructure Patterns?

- 基于自然语言描述,用GPT-5 nano分析多语言代码库中的架构模式。
- 检测F1得分在0.09到0.70之间,依赖模式出现频率和特征明显性。
- 识别常见主导组件的模式更准确,适合架构师与工具开发者参考。
架构模式广泛存在于各类软件制品中。现有工具多仅支持单一语言,难以应对模式多样性和实现差异。大型语言模型(LLMs)因训练数据涵盖多种软件制品与知识,可能突破此限制。为探究其真实效能,我们开发了MicroPAD,利用GPT-5 nano基于自然语言描述,在任意语言的软件制品中识别架构模式。我们选取190个GitHub仓库,通过联系顶级贡献者构建了一个全新的人工标注数据集,聚焦基础设施相关微服务模式。结果表明,MicroPAD可在多语言、多类型制品中检测模式实例,但性能随模式而异(F1分数0.09–0.70),主要受模式出现频率及表现特征显著性影响。关联明显主导组件的模式检测更可靠。这些发现是否适用于其他大模型与工具,是未来研究的重要方向。
原文摘要 · Abstract (English)
Architectural patterns are frequently found in various software artifacts. The wide variety of patterns and their implementations makes detection challenging with current tools, especially since they often only support detecting patterns in artifacts written in a single language. Large Language Models (LLMs), trained on a diverse range of software artifacts and knowledge, might overcome the limitations of existing approaches. However, their true effectiveness and the factors influencing their performance have not yet been thoroughly examined. To better understand this, we developed MicroPAD. This tool utilizes GPT 5 nano to identify architectural patterns in software artifacts written in any language, based on natural-language pattern descriptions. We used MicroPAD to evaluate an LLM's ability to detect instances of architectural patterns, particularly infrastructure-related microservice patterns. To accomplish this, we selected a set of GitHub repositories and contacted their top contributors to create a new, human-annotated dataset of 190 repositories containing microservice architectural patterns. The results show that MicroPAD was capable of detecting pattern instances across multiple languages and artifact types. The detection performance varied across patterns (F1 scores ranging from 0.09 to 0.70), specifically in relation to their prevalence and the distinctiveness of the artifacts through which they manifest. We also found that patterns associated with recognizable, dominant artifacts were detected more reliably. Whether these findings generalize to other LLMs and tools is a promising direction for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。