对比Transformer时代生成与判别模型在文本分类中的表现差异。
Generative or Discriminative? Revisiting Text Classification in the Era of Transformers
- 对比自回归、掩码语言建模等生成式与编码器判别式模型
- 发现不同架构下存在经典‘两阶段’现象，生成模型样本效率更高
- 为数据少或延迟敏感场景提供选型参考
生成与判别分类器的比较自Efron对逻辑回归与判别分析的经典分析以来一直吸引着研究者。早期理论表明，在简单线性设定下，生成模型具有更低的样本复杂度但更高的渐近误差，但这些权衡在Transformer时代尚未被深入探索。本文首次全面评估了现代生成与判别架构——自回归建模、掩码语言建模、离散扩散模型及文本分类编码器。研究发现，经典的‘两阶段’现象在不同架构和训练范式中表现出明显差异。除准确率外，还分析了样本效率、校准性、噪声鲁棒性和序数性等多维度表现。结果为在实际约束如延迟和数据限制下选择最合适建模方法提供了实践指导。
原文摘要 · Abstract (English)
The comparison between discriminative and generative classifiers has intrigued researchers since Efron's seminal analysis of logistic regression versus discriminant analysis. While early theoretical work established that generative classifiers exhibit lower sample complexity but higher asymptotic error in simple linear settings, these trade-offs remain unexplored in the transformer era. We present the first comprehensive evaluation of modern generative and discriminative architectures - Auto-regressive modeling, Masked Language Modeling, Discrete Diffusion, and Encoders for text classification. Our study reveals that the classical 'two regimes' phenomenon manifests distinctly across different architectures and training paradigms. Beyond accuracy, we analyze sample efficiency, calibration, noise robustness, and ordinality across diverse scenarios. Our findings offer practical guidance for selecting the most suitable modeling approach based on real-world constraints such as latency and data limitations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。