自动提取模式生成标注规则，提升软件工程聊天机器人训练效率。
An Approach for Auto Generation of Labeling Functions for Software Engineering Chatbots
- 从已标注查询中自动提取模式生成标注函数。
- 生成的规则使NLU性能最高提升27.2%，AUC达85.3%。
- 适合需快速构建标注数据的开发者与聊天机器人实践者。
软件工程（SE）聊天机器人日益受到关注，其核心是自然语言理解平台（NLUs），需依赖标注数据进行训练。然而，由于高质量数据集稀缺，且训练需特定领域词汇，获取此类数据极具挑战性。以往方法依赖人工设计标注规则（标注函数，LFs），耗时耗力。本文提出一种自动生成LFs的方法，通过从已有标注查询中提取模式实现。在四个SE数据集上评估显示，生成的LFs可使数据标注的AUC最高达85.3%，并使NLU性能提升最多27.2%。结果还表明，标注函数数量影响标注效果。该方法可显著节省标注时间与资源，让开发者更专注于聊天机器人核心功能开发。
原文摘要 · Abstract (English)
Software engineering (SE) chatbots are increasingly gaining attention for their role in enhancing development processes. At the core of chatbots are Natural Language Understanding platforms (NLUs), which enable them to comprehend user queries but require labeled data for training. However, acquiring such labeled data for SE chatbots is challenging due to the scarcity of high-quality datasets, as training requires specialized vocabulary and phrases not found in typical language datasets. Consequently, developers often resort to manually annotating user queries -- a time-consuming and resource-intensive process. Previous approaches require human intervention to generate rules, called labeling functions (LFs), that categorize queries based on specific patterns. To address this issue, we propose an approach to automatically generate LFs by extracting patterns from labeled user queries. We evaluate our approach on four SE datasets and measure performance improvement from training NLUs on queries labeled by the generated LFs. The generated LFs effectively label data with AUC scores up to 85.3% and NLU performance improvements up to 27.2%. Furthermore, our results show that the number of LFs affects labeling performance. We believe that our approach can save time and resources in labeling users' queries, allowing practitioners to focus on core chatbot functionalities rather than manually labeling queries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。