新基准BTZSC对比四类模型零样本文本分类效果,发现重排序器表现最优。
BTZSC: A Benchmark for Zero-Shot Text Classification Across Cross-Encoders, Embedding Models, Rerankers and LLMs
- 构建22个数据集的跨模型零样本分类评测基准
- 重排序器Qwen3-Reranker-8B达0.72宏F1,领先其他模型
- 适合关注零样本分类性能与部署效率的研究者
零样本文本分类(ZSC)通过直接匹配文本与可读标签描述,有望避免昂贵的任务标注。早期方法多依赖微调过的自然语言推理(NLI)交叉编码器,但近年来文本嵌入模型、重排序器和指令微调大模型(LLMs)挑战了这一主导地位。然而,系统性比较仍具挑战,现有评估如MTEB常通过监督探针或微调引入标注样本,未充分考察真正的零样本能力。为此,我们提出BTZSC,涵盖22个公开数据集,覆盖情感、主题、意图和情绪分类,涵盖多样领域、类别数量和文档长度。基于此,我们对四类模型——NLI交叉编码器、嵌入模型、重排序器和指令微调LLMs——进行了系统比较,包含38个公共及自定义检查点。结果表明:(i) 现代重排序器(如Qwen3-Reranker-8B)以0.72宏F1达到新最佳;(ii) GTE-large-en-v1.5等强嵌入模型显著缩小差距,同时在准确率与延迟间取得最优平衡;(iii) 4–12B参数指令微调LLMs表现竞争力(最高宏F1 0.67),尤其在主题分类中优异,但仍落后于专用重排序器;(iv) NLI交叉编码器随主干增大趋于饱和;(v) 扩展主要提升重排序器和LLMs,对嵌入模型增益有限。BTZSC及配套代码已开源,支持零样本理解研究的公平与可复现进展。
原文摘要 · Abstract (English)
Zero-shot text classification (ZSC) offers the promise of eliminating costly task-specific annotation by matching texts directly to human-readable label descriptions. While early approaches have predominantly relied on cross-encoder models fine-tuned for natural language inference (NLI), recent advances in text-embedding models, rerankers, and instruction-tuned large language models (LLMs) have challenged the dominance of NLI-based architectures. Yet, systematically comparing these diverse approaches remains difficult. Existing evaluations, such as MTEB, often incorporate labeled examples through supervised probes or fine-tuning, leaving genuine zero-shot capabilities underexplored. To address this, we introduce BTZSC, a comprehensive benchmark of 22 public datasets spanning sentiment, topic, intent, and emotion classification, capturing diverse domains, class cardinalities, and document lengths. Leveraging BTZSC, we conduct a systematic comparison across four major model families, NLI cross-encoders, embedding models, rerankers and instruction-tuned LLMs, encompassing 38 public and custom checkpoints. Our results show that: (i) modern rerankers, exemplified by Qwen3-Reranker-8B, set a new state-of-the-art with macro F1 = 0.72; (ii) strong embedding models such as GTE-large-en-v1.5 substantially close the accuracy gap while offering the best trade-off between accuracy and latency; (iii) instruction-tuned LLMs at 4--12B parameters achieve competitive performance (macro F1 up to 0.67), excelling particularly on topic classification but trailing specialized rerankers; (iv) NLI cross-encoders plateau even as backbone size increases; and (v) scaling primarily benefits rerankers and LLMs over embedding models. BTZSC and accompanying evaluation code are publicly released to support fair and reproducible progress in zero-shot text understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。