融合自回归与自编码模型，提升小样本文本分类效果
Combining Autoregressive and Autoencoder Language Models for Text Classification
- 用自回归模型生成上下文信息，与原文本结合输入自编码器
- 在4个基准数据集上优于现有方法，小样本任务表现更优
- 适合社会科学研究中样本量小的自动化内容分析场景
本文提出CAALM-TC（结合自回归与自编码语言模型的文本分类），通过融合自回归模型与自编码模型来增强文本分类性能。自回归大模型如GPT、Llama或Phi虽具丰富上下文知识，但文本分类表现通常不及监督微调的BERT模型。CAALM利用自回归模型基于输入文本生成上下文信息，再与原始文本一同输入自编码模型进行分类。该混合方法同时发挥自回归模型的上下文理解优势与自编码模型的高效分类能力。在4个基准数据集上的实验表明，CAALM始终优于现有方法，尤其在小样本及抽象分类任务中表现突出。结果表明，该方法为社会科学中的自动化内容分析提供了可扩展且高效的解决方案，显著降低对标注样本量的需求。
原文摘要 · Abstract (English)
This paper presents CAALM-TC (Combining Autoregressive and Autoencoder Language Models for Text Classification), a novel method that enhances text classification by integrating autoregressive and autoencoder language models. Autoregressive large language models such as Open AI's GPT, Meta's Llama or Microsoft's Phi offer promising prospects for content analysis practitioners, but they generally underperform supervised BERT based models for text classification. CAALM leverages autoregressive models to generate contextual information based on input texts, which is then combined with the original text and fed into an autoencoder model for classification. This hybrid approach capitalizes on the extensive contextual knowledge of autoregressive models and the efficient classification capabilities of autoencoders. Experimental results on four benchmark datasets demonstrate that CAALM consistently outperforms existing methods, particularly in tasks with smaller datasets and more abstract classification objectives. The findings indicate that CAALM offers a scalable and effective solution for automated content analysis in social science research that minimizes sample size requirements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。