arXiv:2503.00673cs.SEcs.LG2025-03被引 3

用大模型提升开发者聊天中的问题清晰度,让代码疑问更容易被解答。

Towards Refining Developer Questions using LLM-Based Named Entity Recognition for Developer Chatroom Conversations

  • 基于大模型构建软件专用实体识别与意图分类框架,精准标注聊天内容。
  • 实体识别准确率达86%,意图检测71%,问题可解决状态判断89%。
  • 发现编程语言、用户变量等要素影响解决率,适合改进开发沟通工具。

在软件工程聊天室中,模糊不清的问题常导致无法回应。识别关键实体对提升问题清晰度和交流效率至关重要,但现有自然语言处理方法常忽略软件领域特异性。本文提出SENIR(软件特定命名实体识别、意图检测与解决状态分类)方法,利用大语言模型标注开发者聊天对话中的实体、意图和解决状态。为量化提升问题清晰度与可解性,构建了基于SENIR标签的解决预测模型,融合额外特征。在DISCO数据集上评估显示:实体识别F-score达86%,意图检测71%,解决状态分类89%。解决预测模型在多种采样策略与验证方式下AUC值介于0.7至0.8之间。正向情绪及‘编程语言’‘用户变量’等实体显著提升解决率,诊断类实体更关乎错误相关问题。不同意图间解决率差异显著:关于API使用与变更的问题更易解决,而差异与评审类问题解决率较低,卡方检验确认其统计显著性。

原文摘要 · Abstract (English)

In software engineering chatrooms, communication is often hindered by imprecise questions that cannot be answered. Recognizing key entities can be essential for improving question clarity and facilitating better exchange. However, existing research using natural language processing techniques often overlooks these software-specific nuances. In this paper, we introduce Software-specific Named Entity Recognition, Intent Detection, and Resolution Classification (SENIR), a labeling approach that leverages a Large Language Model to annotate entities, intents, and resolution status in developer chatroom conversations. To offer quantitative guidance for improving question clarity and resolvability, we build a resolution prediction model that leverages SENIR's entity and intent labels along with additional predictive features. We evaluate SENIR on the DISCO dataset using a subset of annotated chatroom dialogues. SENIR achieves an 86% F-score for entity recognition, a 71% F-score for intent detection, and an 89% F-score for resolution status classification. Furthermore, our resolution prediction model, tested with various sampling strategies (random undersampling and oversampling with SMOTE) and evaluation methods (5-fold cross-validation, 10-fold cross-validation, and bootstrapping), demonstrates AUC values ranging from 0.7 to 0.8. Key factors influencing resolution include positive sentiment and entities such as Programming Language and User Variable across multiple intents, while diagnostic entities are more relevant in error-related questions. Moreover, resolution rates vary significantly by intent: questions about API Usage and API Change achieve higher resolution rates, whereas Discrepancy and Review have lower resolution rates. A Chi-Square analysis confirms the statistical significance of these differences.

大模型代码问答聊天分析意图识别

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。