arXiv:2409.19749cs.CL2024-09EMNLP被引 23

通过最大化主题与预训练模型的互信息,提升主题建模效率与质量。

NeuroMax: Enhancing Neural Topic Modeling via Maximizing Mutual Information and Group Topic Regularization

  • 用互信息约束编码器与预训练模型的主题表示对齐
  • 引入最优传输建模主题间关系,生成更连贯的主题组
  • 推理更快、主题更清晰,适合追求高效精准的主题分析

近年来神经主题模型主要聚焦于两方面:将推理网络(编码器)与预训练语言模型(PLM)结合,以及在生成模型(解码器)中建模词与主题的关系。然而,大规模PLM的使用显著增加了推理开销,不利于低延迟场景。同时,需兼顾词-主题关系与主题-主题关系的建模。本文提出NeuroMax框架,通过最大化神经主题模型编码器所得主题表示与PLM表示之间的互信息,并利用最优传输方法分析主题间信息传递,学习主题间的关联结构。实验表明,NeuroMax在降低推理时间的同时,生成更具连贯性的主题与主题组,并产出更代表性的文档嵌入,在下游任务中表现更优。

原文摘要 · Abstract (English)

Recent advances in neural topic models have concentrated on two primary directions: the integration of the inference network (encoder) with a pre-trained language model (PLM) and the modeling of the relationship between words and topics in the generative model (decoder). However, the use of large PLMs significantly increases inference costs, making them less practical for situations requiring low inference times. Furthermore, it is crucial to simultaneously model the relationships between topics and words as well as the interrelationships among topics themselves. In this work, we propose a novel framework called NeuroMax (Neural Topic Model with Maximizing Mutual Information with Pretrained Language Model and Group Topic Regularization) to address these challenges. NeuroMax maximizes the mutual information between the topic representation obtained from the encoder in neural topic models and the representation derived from the PLM. Additionally, NeuroMax employs optimal transport to learn the relationships between topics by analyzing how information is transported among them. Experimental results indicate that NeuroMax reduces inference time, generates more coherent topics and topic groups, and produces more representative document embeddings, thereby enhancing performance on downstream tasks.

主题建模互信息最优传输高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。