通过分析开发者提问中的情感，高效捕捉隐式满意度信号。
Reading Between the Lines: Scalable User Feedback via Implicit Sentiment in Developer Prompts
- 用自然语言情感分析挖掘开发者提问中的隐式反馈。
- 在372名开发者日志中识别出8%的满意信号，超显式反馈13倍。
- 无需额外标注，适合大规模评估开发体验。
大规模评估对话式AI助手对开发者的满意度至关重要但极具挑战。用户研究虽深入，却难以扩展；而来自日志或产品内评分的量化信号通常过于浅显或稀疏，不可靠。为填补这一空白，我们提出并验证了一种新方法：通过分析开发者提问中的情感来识别隐式满意度信号。基于对372名专业开发者工业级使用日志的分析，该方法可识别约8%的交互中存在满意信号，效率是显式反馈的13倍以上，且即使采用现成的情感分析工具也具备合理准确性。这一实用方法可补充现有反馈渠道，为大规模理解开发者体验开辟新路径。
原文摘要 · Abstract (English)
Evaluating developer satisfaction with conversational AI assistants at scale is critical but challenging. User studies provide rich insights, but are unscalable, while large-scale quantitative signals from logs or in-product ratings are often too shallow or sparse to be reliable. To address this gap, we propose and evaluate a new approach: using sentiment analysis of developer prompts to identify implicit signals of user satisfaction. With an analysis of industrial usage logs of 372 professional developers, we show that this approach can identify a signal in ~8% of all interactions, a rate more than 13 times higher than explicit user feedback, with reasonable accuracy even with an off-the-shelf sentiment analysis approach. This new practical approach to complement existing feedback channels would open up new directions for building a more comprehensive understanding of the developer experience at scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。