用比赛机制让大模型自动排序，提升零样本分类效果。
Using tournaments to calculate AUROC for zero-shot classification with LLMs
- 将分类转为实例间两两比较，用评分系统生成置信度排序。
- 新算法减少比较次数，性能优于传统零样本方法。
- 适合需要可解释排序的零样本场景，如医疗、金融决策。
大型语言模型在许多零样本分类任务中表现惊人，但因缺乏可调节的决策边界，难以与监督分类器公平比较。本文提出一种方法，将二分类任务转化为数据集中实例间的成对比较，利用大模型生成这些实例的相对排序。通过重复成对比较，采用类似国际象棋的Elo评分系统为实例打分，从而建立数据集内实例的置信度排序。我们评估了多种调度算法以最小化比较次数，结果表明所提算法不仅减少比较量，还提升了分类性能，同时提供比传统零样本分类更丰富的信息。
原文摘要 · Abstract (English)
Large language models perform surprisingly well on many zero-shot classification tasks, but are difficult to fairly compare to supervised classifiers due to the lack of a modifiable decision boundary. In this work, we propose and evaluate a method that transforms binary classification tasks into pairwise comparisons between instances within a dataset, using LLMs to produce relative rankings of those instances. Repeated pairwise comparisons can be used to score instances using the Elo rating system (used in chess and other competitions), inducing a confidence ordering over instances in a dataset. We evaluate scheduling algorithms for their ability to minimize comparisons, and show that our proposed algorithm leads to improved classification performance, while also providing more information than traditional zero-shot classification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。