arXiv:2504.20781cs.SEcs.AI2025-04中稿 · publication in ACM…被引 11

用大模型自动生成软件架构决策理由,提升开发文档质量。

Using LLMs in Generating Design Rationale for Software Architecture Decisions

  • 用三种提示策略让大模型生成架构决策理由。
  • 生成内容召回率62.7%~71.5%,准确率约27%。
  • 生成理由中有近七成有参考价值,适合开发者辅助写作。

软件架构决策的设计理由于提供架构选择背后的推理而具有重要价值,但实践中常因开发者缺乏动力而记录不足。随着大语言模型(LLMs)的发展,其在文本理解、推理与生成方面的能力为自动生成或恢复设计理由提供了可能。本研究评估了五种LLMs在生成架构决策设计理由上的表现,基于100个来自Stack Overflow、GitHub Issues和GitHub Discussions的架构相关问题构建数据集,采用零样本、思维链(CoT)及基于LLM的代理三种提示策略。以人类专家提供的设计理由为基准,生成结果的精确率在0.267至0.278之间,召回率在0.627至0.715之间,F1分数为0.351至0.389。此外,64.45%至69.42%的非专家提及论点具有实用价值,4.12%至4.87%存在正确性不确定,1.59%至3.24%可能误导。通过六名实践者的半结构化访谈,进一步探讨了不同提示策略的优劣、生成内容的可靠性及其实际应用意义。

原文摘要 · Abstract (English)

Design Rationale (DR) for software architecture decisions refers to the reasoning underlying architectural choices, which provides valuable insights into the different phases of the architecting process throughout software development. However, in practice, DR is often inadequately documented due to a lack of motivation and effort from developers. With the recent advancements in Large Language Models (LLMs), their capabilities in text comprehension, reasoning, and generation may enable the generation and recovery of DR for architecture decisions. In this study, we evaluated the performance of LLMs in generating DR for architecture decisions. First, we collected 50 Stack Overflow (SO) posts, 25 GitHub issues, and 25 GitHub discussions related to architecture decisions to construct a dataset of 100 architecture-related problems. Then, we selected five LLMs to generate DR for the architecture decisions with three prompting strategies, including zero-shot, chain of thought (CoT), and LLM-based agents. With the DR provided by human experts as ground truth, the Precision of LLM-generated DR with the three prompting strategies ranges from 0.267 to 0.278, Recall from 0.627 to 0.715, and F1-score from 0.351 to 0.389. Additionally, 64.45% to 69.42% of the arguments of DR not mentioned by human experts are also helpful, 4.12% to 4.87% of the arguments have uncertain correctness, and 1.59% to 3.24% of the arguments are potentially misleading. To further understand the trustworthiness and applicability of LLM-generated DR in practice, we conducted semi-structured interviews with six practitioners. Based on the experimental and interview results, we discussed the pros and cons of the three prompting strategies, the strengths and limitations of LLM-generated DR, and the implications for the practical use of LLM-generated DR.

大模型架构决策设计理由代码文档

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。