提出更可靠的评估方法，解决主题模型结果不一致的问题。
Reliability of Topic Modeling
- 基于测量理论，提出三种新可靠性评估指标。
- 实验证明麦当劳ω系数最能反映主题模型的稳定性。
- 适合依赖主题模型的学术研究者用于方法验证。
主题模型可从文本数据中提取潜在因子，并用于后续统计分析，但其结果常因初始化差异、采样随机性或数据噪声而显著变化。由于许多研究将学习到的主题模型视为真实变量进行分析，其可靠性尤为关键。本文指出，当前衡量主题模型可靠性的标准做法未能捕捉两种常用模型中的关键变异。基于测量理论，我们对三种新指标进行了理论与实证分析。在合成数据和真实数据上，麦当劳ω（McDonald's ω）表现最优，能最准确地表征主题模型的可靠性。该指标为验证主题模型方法提供了必要工具，应成为所有基于主题模型研究的标准组成部分。
原文摘要 · Abstract (English)
Topic models allow researchers to extract latent factors from text data and use those variables in downstream statistical analyses. However, these methodologies can vary significantly due to initialization differences, randomness in sampling procedures, or noisy data. Reliability of these methods is of particular concern as many researchers treat learned topic models as ground truth for subsequent analyses. In this work, we show that the standard practice for quantifying topic model reliability fails to capture essential aspects of the variation in two widely-used topic models. Drawing from a extensive literature on measurement theory, we provide empirical and theoretical analyses of three other metrics for evaluating the reliability of topic models. On synthetic and real-world data, we show that McDonald's $ω$ provides the best encapsulation of reliability. This metric provides an essential tool for validation of topic model methodologies that should be a standard component of any topic model-based research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。