arXiv:2504.08207cs.SEcs.AI2025-04被引 6

用大模型自动生成高质量架构决策记录,兼顾隐私与效率。

DRAFT-ing Architectural Design Decisions using LLMs

  • 结合检索增强与微调的两阶段方法,提升生成质量
  • 在4911条架构记录上验证,优于现有所有方法
  • 适合需要隐私保护的软件架构团队使用

架构知识管理对软件开发至关重要,但因缺乏标准化和人工成本高而困难。架构决策记录(ADR)虽能结构化记录架构设计决策(ADD),但因手动编写费力且工具支持不足,应用受限。此前研究显示大语言模型(LLM)可辅助生成ADD,但直接提示效果不佳。使用第三方LLM存在隐私风险,自托管则资源消耗大。为此,我们实验了少样本、检索增强生成(RAG)和微调等方法,发现均能提升效果。在此基础上,提出领域特定检索增强少样本微调方法(DRAFT),融合三种技术优势。DRAFT分两阶段:离线阶段在检索到的示例上微调模型,线上阶段利用检索到的ADR与微调模型生成ADD。我们在包含4,911条ADR的数据集上,对多种LLM进行评估,采用自动化指标与人工评价。结果表明,DRAFT在有效性上全面超越其他方法,同时保持高效。研究证明DRAFT可帮助架构师高效生成决策记录,同时解决隐私与资源问题。

原文摘要 · Abstract (English)

Architectural Knowledge Management (AKM) is crucial for software development but remains challenging due to the lack of standardization and high manual effort. Architecture Decision Records (ADRs) provide a structured approach to capture Architecture Design Decisions (ADDs), but their adoption is limited due to the manual effort involved and insufficient tool support. Our previous work has shown that Large Language Models (LLMs) can assist in generating ADDs. However, simply prompting the LLM does not produce quality ADDs. Moreover, using third-party LLMs raises privacy concerns, while self-hosting them poses resource challenges. To this end, we experimented with different approaches like few-shot, retrieval-augmented generation (RAG) and fine-tuning to enhance LLM's ability to generate ADDs. Our results show that both techniques improve effectiveness. Building on this, we propose Domain Specific Retreival Augumented Few Shot Fine Tuninng, DRAFT, which combines the strengths of all these three approaches for more effective ADD generation. DRAFT operates in two phases: an offline phase that fine-tunes an LLM on generating ADDs augmented with retrieved examples and an online phase that generates ADDs by leveraging retrieved ADRs and the fine-tuned model. We evaluated DRAFT against existing approaches on a dataset of 4,911 ADRs and various LLMs and analyzed them using automated metrics and human evaluations. Results show DRAFT outperforms all other approaches in effectiveness while maintaining efficiency. Our findings indicate that DRAFT can aid architects in drafting ADDs while addressing privacy and resource constraints.

架构设计大模型知识管理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。