测试大模型处理长文本分类表现，发现通用模型优于专为长文本设计的模型。
Beyond Token Limits: Assessing Language Model Performance on Long Text Classification
- 对比多种模型在长文本分类任务中的表现，涵盖5种语言。
- 开源模型在21类政策主题分类中表现优于GPT-3.5和GPT-4。
- 模型性能受类别间支持与内容重叠程度影响，适合政策分析研究者参考。
社会科学研究中广泛使用的大型语言模型（如BERT及其衍生模型RoBERTa）存在输入文本长度限制，这在处理长文本分类任务时尤为突出。本文针对法律草案等可达数百页的长文本，使用XLM-RoBERTa、Longformer、GPT-3.5和GPT-4在21个政策主题标签（涵盖教育至医疗保健）的多分类任务上展开实验，覆盖5种语言。结果显示，专为长文本设计的Longformer并未表现出显著优势；相比之下，最佳开源模型在性能上超越了GPT系列变体。进一步分析表明，特定类别间的支持度与内容重叠程度是影响长文本分类效果的关键因素。
原文摘要 · Abstract (English)
The most widely used large language models in the social sciences (such as BERT, and its derivatives, e.g. RoBERTa) have a limitation on the input text length that they can process to produce predictions. This is a particularly pressing issue for some classification tasks, where the aim is to handle long input texts. One such area deals with laws and draft laws (bills), which can have a length of multiple hundred pages and, therefore, are not particularly amenable for processing with models that can only handle e.g. 512 tokens. In this paper, we show results from experiments covering 5 languages with XLM-RoBERTa, Longformer, GPT-3.5, GPT-4 models for the multiclass classification task of the Comparative Agendas Project, which has a codebook of 21 policy topic labels from education to health care. Results show no particular advantage for the Longformer model, pre-trained specifically for the purposes of handling long inputs. The comparison between the GPT variants and the best-performing open model yielded an edge for the latter. An analysis of class-level factors points to the importance of support and substance overlaps between specific categories when it comes to performance on long text inputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。