arXiv:2502.01270cs.CL2025-02NAACL被引 2

通过主谓宾结构自动标注意图分类解释信号,提升模型可解释性。

Main Predicate and Their Arguments as Explanation Signals For Intent Classification

  • 基于主谓宾结构自动标注解释信号,无需人工标注
  • 在ATIS和SNIPS数据集上构建21,000条带解释标注样本
  • 引导模型关注解释信号后,推理合理性提升3-4%

意图分类对对话系统至关重要,深度学习模型表现良好,但可解释性研究受限于缺乏合适基准数据。人工标注解释信号耗时且成本高。观察发现,文本中的主谓动词常表示动作,直接宾语常指示对话领域,可作为解释信号。据此提出新方法:自动为意图分类数据集中的文本样本添加词级解释,标记主谓动词及其依存关系作为解释信号,在ATIS和SNIPS数据集上构建了包含21,000个实例的唯一可解释性数据集。进一步实验表明,虽高性能分类模型在可解释性指标(如合理性和忠实性)上表现不佳,但通过在训练中引导模型关注本数据集中的解释信号,可使Token F1评分提升3-4%,显著改善模型推理能力。

原文摘要 · Abstract (English)

Intent classification is crucial for conversational agents (chatbots), and deep learning models perform well in this area. However, little research has been done on the explainability of intent classification due to the absence of suitable benchmark data. Human annotation of explanation signals in text samples is time-consuming and costly. However, from inspection of data on intent classification, we see that, more often than not, the main verb denotes the action, and the direct object indicates the domain of conversation, serving as explanation signals for intent. This observation enables us to hypothesize that the main predicate in the text utterances, along with the arguments of the main predicate, can serve as explanation signals. Leveraging this, we introduce a new technique to automatically augment text samples from intent classification datasets with word-level explanations. We mark main predicates (primarily verbs) and their arguments (dependency relations) as explanation signals in benchmark intent classification datasets ATIS and SNIPS, creating a unique 21k-instance dataset for explainability. Further, we experiment with deep learning and language models. We observe that models that work well for classification do not perform well in explainability metrics like plausibility and faithfulness. We also observe that guiding models to focus on explanation signals from our dataset during training improves the plausibility Token F1 score by 3-4%, improving the model's reasoning.

意图分类可解释性自然语言处理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。