用语言模型软标签提升主题模型，让主题更准更相关。
Improving Topic Modeling by Distilling Soft Labels from Language Models
- 从语言模型获取上下文软标签，指导主题模型训练
- 在多个数据集上主题连贯性与分配准确率显著提升
- 适合需要精准主题提取和文档检索的应用场景
传统神经主题模型通常通过重建文档的词袋表示进行优化，忽略了上下文信息且在数据稀疏时表现不佳。本文提出一种新的主题模型训练框架——从语言模型中蒸馏软标签（DSL）。通过将特定提示条件下下一个词的概率投影到预定义词汇表，并利用语言模型隐藏状态训练主题模型重建软标签，生成更具上下文意义的重构信号。实验表明，该方法在多个基准上显著提升了主题连贯性和主题分配准确率。此外，我们引入一种基于检索的评估指标，结果显示本方法在识别语义相似文档方面明显优于现有方法，验证了其在检索导向应用中的有效性。
原文摘要 · Abstract (English)
Traditional neural topic models are typically optimized by reconstructing the document's Bag-of-Words (BoW) representations, overlooking contextual information and struggling with data sparsity. In this work, we introduce a novel topic model training framework by Distilling Soft Labels (DSL) from Language Models (LMs). To construct the contextually enriched reconstruction signals, we project the next token probabilities, conditioned on a specialized prompt, onto a pre-defined vocabulary, and train the topic models to reconstruct the soft labels using the LM hidden states. This produces higher-quality topics that are more closely aligned with the underlying thematic structure of the corpus. Extensive experiments demonstrate that DSL achieves substantial improvements in topic coherence and assignment accuracy over existing baselines. Additionally, we also introduce a retrieval-based metric, which shows that our approach significantly outperforms existing methods in identifying semantically similar documents, highlighting its effectiveness for retrieval-oriented applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。