arXiv:2507.15742cs.CLcs.IR2025-07中稿 · The American Stati…被引 7

用费雪精确检验解释TF-IDF为何有效,为经典算法提供统计学根基。

A Fisher's exact test justification of the TF-IDF term-weighting scheme

  • 将TF-ICF与单尾费雪精确检验的p值负对数关联起来。
  • 在理想假设下,TF-IDF与负对数p值存在数学等价关系。
  • 适用于需要理论解释的统计学者和信息检索研究者。

词频-逆文档频率(TF-IDF)是信息检索领域最著名的数学表达式之一。尽管它最初被视为一种简单的启发式方法,用于衡量特定词在某文档中出现的集中程度,但其众多变体仍广泛应用于各类文本分析任务中。本文致力于为TF-IDF建立坚实的理论基础,通过统计学视角揭示其合理性:在适度正则条件下,常见的TF-ICF变体与单尾费雪精确检验的负对数p值高度相关。作为推论,在理想化假设下,TF-IDF与该负对数p值存在明确联系。进一步证明,在文档集合趋于无限大的极限情况下,该统计量收敛于标准的TF-IDF。这一费雪精确检验的解释为统计学家提供了对长期有效的词权重方案的合理说明。

原文摘要 · Abstract (English)

Term frequency-inverse document frequency, or TF-IDF for short, is arguably the most celebrated mathematical expression in the history of information retrieval. Conceived as a simple heuristic quantifying the extent to which a given term's occurrences are concentrated in any one given document out of many, TF-IDF and its many variants are routinely used as term-weighting schemes in diverse text analysis applications. There is a growing body of scholarship dedicated to placing TF-IDF on a sound theoretical foundation. Building on that tradition, this paper justifies the use of TF-IDF to the statistics community by demonstrating how the famed expression can be understood from a significance testing perspective. We show that the common TF-IDF variant TF-ICF is, under mild regularity conditions, closely related to the negative logarithm of the $p$-value from a one-tailed version of Fisher's exact test of statistical significance. As a corollary, we establish a connection between TF-IDF and the said negative log-transformed $p$-value under certain idealized assumptions. We further demonstrate, as a limiting case, that this same quantity converges to TF-IDF in the limit of an infinitely large document collection. The Fisher's exact test justification of TF-IDF equips the working statistician with a ready explanation of the term-weighting scheme's long-established effectiveness.

TF-IDF统计检验信息检索词权重

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。