用文本变异检测大模型分类置信度，提升零样本文本分类效果
PCS: Perceived Confidence Scoring of Black Box LLMs with Metamorphic Relations
- 通过语义等价的文本变异生成多个版本，检验模型输出一致性
- 在文本分类任务中使零样本模型性能提升9.3%，多模型投票时提升5.8%
- 适用于单模型或多个模型协同场景，无需额外训练
零样本大语言模型现被用于文本分类任务，如句子或文章的情感与偏见检测。然而其在数据标注任务中的表现可能不佳。本文提出一种新方法，通过利用元变换关系（Metamorphic Relations, MRs）评估大模型对文本输入分类的置信度。MRs生成语义等价但文本形式不同的输入变体，根据元测试原则，这些变体应具有相似的标注标签。通过分析模型在这些变体上的响应一致性，计算感知置信度分数（PCS），该分数基于预测标签的出现频率。PCS可应用于单模型或多模型设置（如多数投票）。实证评估表明，基于PCS的方法使零样本大模型在文本分类任务中性能提升9.3%；在多模型多数投票设置下，性能提升5.8%。
原文摘要 · Abstract (English)
Zero-shot LLMs are now also used for textual classification tasks, e.g., sentiment and bias detection in a sentence or article. However, their performance can be suboptimal in such data annotation tasks. We introduce a novel technique that evaluates an LLM's confidence for classifying a textual input by leveraging Metamorphic Relations (MRs). The MRs generate semantically equivalent yet textually divergent versions of the input. Following the principles of Metamorphic Testing (MT), the mutated versions are expected to have annotation labels similar to the input. By analyzing the consistency of an LLM's responses across these variations, we compute a perceived confidence score (PCS) based on the frequency of the predicted labels. PCS can be used for both single and multiple LLM settings (e.g., when multiple LLMs are vetted in a majority-voting setup). Empirical evaluation shows that our PCS-based approach improves the performance of zero-shot LLMs by 9.3% in textual classification tasks. When multiple LLMs are used in a majority-voting setup, we obtain a performance boost of 5.8% with PCS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。