通过分析用户真实评价，揭示开发者对编程助手的期待与痛点。
"My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding Assistants
- 基于32款插件的用户评论，构建使用反馈分类体系。
- 90%的编程助手在近两年发布，但仅占插件总数1.64%。
- 用户不仅要智能建议，更需上下文感知、可定制和低资源消耗。
本文探究人工智能编程助手（如GitHub Copilot）广泛使用背景下开发者的真实需求与批评点，揭示其在实际开发中的期望与挑战。不同于以往在受控环境中的观察研究，本研究分析了来自Visual Studio Code Marketplace的大量一手用户评论，涵盖1,085个AI编程助手——虽仅占所有扩展的1.64%，但其中超过90%在过去两年内发布。通过对32款安装量与评论量充足的助手进行人工抽样分析，构建了用户关切与反馈的完整分类体系，并标注每条评论对特定功能的态度，获得关于功能满意度与不满意度的细致洞察。研究发现，用户不仅期待智能代码建议，更重视上下文感知、可定制性与资源效率。据此提出五项实践建议，以指导编程助手的改进方向。
原文摘要 · Abstract (English)
This paper aims to explore fundamental questions in the era when AI coding assistants like GitHub Copilot are widely adopted: what do developers truly value and criticize in AI coding assistants, and what does this reveal about their needs and expectations in real-world software development? Unlike previous studies that conduct observational research in controlled and simulated environments, we analyze extensive, first-hand user reviews of AI coding assistants, which capture developers' authentic perspectives and experiences drawn directly from their actual day-to-day work contexts. We identify 1,085 AI coding assistants from the Visual Studio Code Marketplace. Although they only account for 1.64% of all extensions, we observe a surge in these assistants: over 90% of them are released within the past two years. We then manually analyze the user reviews sampled from 32 AI coding assistants that have sufficient installations and reviews to construct a comprehensive taxonomy of user concerns and feedback about these assistants. We manually annotate each review's attitude when mentioning certain aspects of coding assistants, yielding nuanced insights into user satisfaction and dissatisfaction regarding specific features, concerns, and overall tool performance. Built on top of the findings-including how users demand not just intelligent suggestions but also context-aware, customizable, and resource-efficient interactions-we propose five practical implications and suggestions to guide the enhancement of AI coding assistants that satisfy user needs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。