实证研究人机协作写代码的高效策略,发现关键交互特征与错误类型。
Experimental Analysis of Productive Interaction Strategy with ChatGPT: User Study on Function and Project-level Code Generation Tasks
- 通过用户实验分析人机交互中的15项特征,找出影响编码效率的关键因素。
- 发现3个显著影响生产力的交互特征,提出5条提升效率的实用指南。
- 梳理29类运行时和逻辑错误,提供针对性应对方案,适合开发者参考。
大型语言模型(LLMs)在软件工程任务中的应用日益广泛,但现有研究多局限于函数级任务和常见提示模式,难以覆盖项目级复杂场景(如跨类依赖)。本文设计了两个项目级基准任务,突破传统局限。通过36名来自不同背景的参与者使用GPT助手完成任务,结合屏幕录制与聊天日志分析其交互行为。研究表明:(1)15项人机交互特征中,有3项显著影响编码生产力;(2)提炼出5条提升交互效率的核心指导原则;(3)构建包含29类运行时与逻辑错误的分类体系,并给出缓解建议。该研究为真实开发场景下的高效人机协作提供了实证依据。
原文摘要 · Abstract (English)
The application of Large Language Models (LLMs) is growing in the productive completion of Software Engineering tasks. Yet, studies investigating the productive prompting techniques often employed a limited problem space, primarily focusing on well-known prompting patterns and mainly targeting function-level SE practices. We identify significant gaps in real-world workflows that involve complexities beyond class-level (e.g., multi-class dependencies) and different features that can impact Human-LLM Interactions (HLIs) processes in code generation. To address these issues, we designed an experiment that comprehensively analyzed the HLI features regarding the code generation productivity. Our study presents two project-level benchmark tasks, extending beyond function-level evaluations. We conducted a user study with 36 participants from diverse backgrounds, asking them to solve the assigned tasks by interacting with the GPT assistant using specific prompting patterns. We also examined the participants' experience and their behavioral features during interactions by analyzing screen recordings and GPT chat logs. Our statistical and empirical investigation revealed (1) that three out of 15 HLI features significantly impacted the productivity in code generation; (2) five primary guidelines for enhancing productivity for HLI processes; and (3) a taxonomy of 29 runtime and logic errors that can occur during HLI processes, along with suggested mitigation plans.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。