测试7种Transformer模型发现，小模型也能达到大模型的文本主题质量。
A comparative study of transformer-based embeddings for topic coherence
- 用BERTopic框架对比7种不同规模的Transformer模型
- 从2200万到130亿参数，模型大小对主题质量影响不大
- 适合关注效率与性能平衡的NLP研究者
主题建模是自然语言处理中将大量文本按词共现模式组织成连贯主题组的方法，其中潜在狄利克雷分配（LDA）仍是应用最广且可解释性高的概率方法。近年来，基于Transformer的语言模型提供了更优的文档表示。研究表明，模型规模（参数量）对语言模型在各类任务上的表现有显著影响。本研究系统评估了模型规模对主题质量的影响，通过在多种语料库上使用BERTopic流程分析七种Transformer模型（从MiniLM等小模型到LLaMA-2等大模型）的表现。主题质量采用Röder等人（2015）提出的一致性与分歧度指标进行评价。结果表明，模型参数量从2200万至130亿之间变化时，对主题质量无明显影响，说明小模型可实现与大模型相当的效果。
原文摘要 · Abstract (English)
Topic modeling is a branch of Natural Language Processing (NLP) that aims to organize large collections of texts into coherent groups according to word co-occurrence patterns, with Latent Dirichlet Allocation (LDA) remaining one of the most widely used and interpretable probabilistic approaches. Recent advances in NLP, particularly transformer-based language models, offer improved document representations. It is also known that the size of the model (in terms of number of parameters) has a significant impact in the performance of the language models on different pre-defined tasks. In this study, we systematically examine the effect of model size on topic quality by analyzing the performances of seven transformer-based language models (from small models such as MiniLM to large ones such as LLaMA-2) in a BERTopic pipeline on a variety of corpora. Topic quality is evaluated using coherence and divergence metrics following R{ö}der et al. (2015). Our results indicate that model size, ranging from 22 million to 13 billion parameters, has a negligible impact on the quality of the topic, suggesting that smaller models can achieve comparable performance to larger models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。