提出文本开放集分类的不确定性估计方法，提升系统拒识准确率。
Uncertainty Estimation for the Open-Set Text Classification systems
- 融合查询与数据分布双重不确定性，改进文本分类拒识能力
- 在多个数据集上提升预测拒识率365%至40%
- 适合需要高可信度文本识别的应用场景
准确的不确定性估计对构建鲁棒可信的识别系统至关重要。本文研究开放集文本分类（OSTC）中的不确定性估计问题：文本样本应被归为已有类别或拒绝为未知。针对OSTC中出现的不同类型不确定性，我们将整体不确定性估计（HolUE）方法拓展至文本领域。该方法同时捕捉因查询表述不清导致的文本不确定性，以及因数据分布模糊引起的图库不确定性，从而有效预测系统出错时刻。我们构建了一个新的OSTC基准，基于作者归属、意图和主题分类数据集进行广泛实验。结果显示，相较于基于质量的SCF基线，HolUE在各数据集上预测拒识率（PRR）提升365%至40%：在Yahoo Answers上达365%（FPIR 0.1时0.79 vs 0.17），DBPedia上347%（0.85 vs 0.19），PAN作者归属任务240%（FPIR 0.5时0.51 vs 0.15），CLINC150意图分类任务40%（0.73 vs ~0.52）。代码与协议已开源。
原文摘要 · Abstract (English)
Accurate uncertainty estimation is essential for building robust and trustworthy recognition systems. In this paper, we consider the open-set text classification (OSTC) task - and uncertainty estimation for it. For OSTC a text sample should be classified as one of the existing classes or rejected as unknown. To account for the different uncertainty types encountered in OSTC, we adapt the Holistic Uncertainty Estimation (HolUE) method for the text domain. Our approach addresses two major causes of prediction errors in text recognition systems: text uncertainty that stems from ill formulated queries and gallery uncertainty that is related the ambiguity of data distribution. By capturing these sources, it becomes possible to predict when the system will make a recognition error. We propose a new OSTC benchmark and conduct extensive experiments on a wide range of data, utilizing the authorship attribution, intent and topic classification datasets. HolUE achieves 40-365% improvement in Prediction Rejection Ratio (PRR) over the quality-based SCF baseline across datasets: 365% on Yahoo Answers (0.79 vs 0.17 at FPIR 0.1), 347% on DBPedia (0.85 vs 0.19), 240% on PAN authorship attribution (0.51 vs 0.15 at FPIR 0.5), and 40% on CLINC150 intent classification (0.73 vs~0.52). We make public our code and protocols https://github.com/Leonid-Erlygin/text_uncertainty.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。