微小干预也能提升开发者与LLM对话体验,效果可期。
On the Prospects of Dynamic LLM Conversations in Software Development

- 用极简方式增强对话,仅调整提示和建议
- 主动引导组满意度明显上升,无负面影响
- 适合想优化编程助手交互的开发者与研究者
大型语言模型(LLMs)已成为开发者的得力助手,但如何有效支持其在开发过程中的交互仍缺乏深入理解。当前与基于聊天的LLM的交互质量高度依赖于开发者提问的方式和信息输入。本文通过为期四个月的纵向研究,考察对开发者-LLM交互进行干预的影响。三组计算机科学专业学生在全栈网页开发项目中使用基于聊天的LLM:(1) 上下文感知组接收基于意图的对话增强;(2) 主动引导组获得后续建议和定制化指导;(3) 对照组无任何干预。干预设计极简,以减少混杂因素并隔离处理效应。分析交互日志与用户调查发现,各组交互模式无显著差异,表明干预未带来可检测的负面影响;同时,主动引导组显示出更高的满意度趋势。结果表明,即使采用最小干预,动态引导机制也已产生可观测影响,更强烈干预可能显著提升开发者满意度。
原文摘要 · Abstract (English)
Large language models (LLMs) have become an essential tool for assisting developers, yet we still lack knowledge on ways to effectively support their interactions during development activities. That is, the quality of interactions with a chat-based LLM still strongly depends on how developers phrase prompts and which information they include. Our goal is to evaluate whether interventions into these interactions with LLMs have an effect on software developers---be it harmful or beneficial. To this end, we conducted a four-month longitudinal study with third-semester computer science students working on a full-stack Web development project using chat-based LLMs under three conditions: (1) a \emph{context}-aware group received intent-based conversation augmentation, (2) a \emph{proactive} group received follow-up suggestions and tailored advice, and (3) a \emph{control} group without intervention. Our augmentations are minimal: (i) to reduce confounding factors and (ii) to isolate treatment effects. Analyzing interaction logs and user surveys revealed no major differences in interaction patterns, indicating no detectable harmful effects in the measured outcomes when intervening in interactions. Moreover, we observed trends of increased satisfaction with the \emph{proactive} treatment. The results indicate that even with minimal interventions, dynamic guidance mechanisms for developer-LLM interactions show observable effects, such that more severe augmentations may have the potential to substantially improve developer satisfaction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。