实证研究开发者如何用大模型辅助软件设计，揭示其用途与局限。
Using LLMs in Software Design: An Empirical Study of GitHub and A Practitioner Survey
- 结合开源代码对话与从业者调研，分析大模型在设计中的应用。
- 发现九类支持的设计任务，多数集中在详细设计与代码生成。
- 亮点：首次系统揭示大模型在设计中的真实使用场景与痛点。
大型语言模型（LLMs）在软件工程多个任务中展现出巨大潜力，包括传统上依赖人类专业知识的软件设计。然而，关于大模型如何被用于软件设计及其优劣的研究仍十分有限。本文通过混合方法研究，分析了GitHub上291个开发者与ChatGPT的对话，并对65名从业者进行了调查。结果揭示了九类由ChatGPT支持的设计任务，包括架构设计、数据模型设计及设计模式应用。开发者主要利用ChatGPT获取知识和生成设计相关代码，多数任务处于详细设计层面。研究识别出七项关键益处，如更优的技术选型和早期发现设计缺陷；同时发现六项限制，包括输出过长难读、生成不可执行或错误代码，以及对上下文依赖导致幻觉。这些发现基于开源与实践双视角，为未来有效整合大模型到设计流程提供了实证基础。
原文摘要 · Abstract (English)
Recent advancements in Large Language Models (LLMs) have demonstrated significant potential across a wide range of software engineering tasks, including software design, an area traditionally regarded as highly dependent on human expertise and judgment. However, there has been little research focusing on how LLMs are used in software design, nor on the associated benefits and drawbacks. This paper aims to bridge this gap by empirically investigating how software developers utilize LLMs in the context of software design. We conduct a mixed-methods study, combining a mining study of 291 developer-ChatGPT conversations shared on GitHub with a survey of 65 software practitioners. Our findings reveal nine distinct categories of design tasks supported by ChatGPT, including architecture design, data model design, and the use of design patterns. We further characterize developer-ChatGPT interactions, showing that developers primarily use ChatGPT for knowledge acquisition and design-related code generation, with most tasks situated at the detailed design level. The study identifies seven key benefits of utilizing LLMs in software design as perceived by developers, such as better technology selection and the early detection of design flaws. We also uncover six limitations, including the generation of overly lengthy and difficult-to-read outputs, the creation of inexecutable or incorrect code, and a heavy reliance on context that can lead to hallucinated results. These findings provide an evidence-based characterization of current LLM use in software design from both open-source and practitioner perspectives, highlighting a tension between perceived benefits and limitations, which lays a foundation for future research and the development of effective techniques and tools to integrate LLMs into software design practices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。