用大模型精简BERTopic生成的冗余话题，提升社交媒体文本的主题质量。
LLM-Assisted Topic Reduction for BERTopic on Social Media Data
- 先用BERTopic生成初始话题，再用大模型迭代合并语义相似的话题。
- 在三个推特数据集上验证，显著提升主题多样性和一致性。
- 适合需要高质量主题分析的社交网络研究者，尤其关注可扩展性。
BERTopic框架利用Transformer嵌入和层次聚类从非结构化文本中提取潜在主题。尽管有效，但在噪声多、稀疏性强的社交媒体数据上常生成过多重叠主题。近期研究尝试使用大语言模型实现端到端主题建模，但通常计算开销大，难以在大数据场景中扩展。本文提出一种结合BERTopic生成主题与大语言模型进行主题精简的框架：首先生成初始主题并构建其表征，再将这些表征输入语言模型，由其迭代识别并合并语义相似的主题。我们在三个Twitter/X数据集和四种不同语言模型上进行了评估，结果表明该方法在提升主题多样性方面优于基线，在多数情况下也增强了主题连贯性，但对数据特征和初始参数选择有一定敏感性。
原文摘要 · Abstract (English)
The BERTopic framework leverages transformer embeddings and hierarchical clustering to extract latent topics from unstructured text corpora. While effective, it often struggles with social media data, which tends to be noisy and sparse, resulting in an excessive number of overlapping topics. Recent work explored the use of large language models for end-to-end topic modelling. However, these approaches typically require significant computational overhead, limiting their scalability in big data contexts. In this work, we propose a framework that combines BERTopic for topic generation with large language models for topic reduction. The method first generates an initial set of topics and constructs a representation for each. These representations are then provided as input to the language model, which iteratively identifies and merges semantically similar topics. We evaluate the approach across three Twitter/X datasets and four different language models. Our method outperforms the baseline approach in enhancing topic diversity and, in many cases, coherence, with some sensitivity to dataset characteristics and initial parameter selection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。