arXiv:2511.04901cs.IRcs.CL2025-11

用熵减方法提升文档关联度判断,比传统tf-idf更精准。

Association via Entropy Reduction

  • 基于熵减原理设计新评分方法aver,自动识别无关配对。
  • aver在真实关联数据上优于tf-idf,能区分相同tf-idf得分的文档对。
  • 适用于大规模文档集合,且理论基础更自然,适合图结构分析场景。

在神经网络兴起之前,词频-逆文档频率(tf-idf)被认为是判断文档与查询相关性的最佳方法。本文提出一种新评分方法aver,并在具有真实关联标记的数据集上验证,aver在发现相关文档对方面表现优于tf-idf。该任务涉及在大型图中识别相关顶点,这可能是当前神经网络尚未占据明显优势的领域。除了这一实例,我们观察到:(1) aver具有天然阈值以判定配对为无关,而tf-idf无此机制;(2) aver可区分tf-idf均得分为1.0的文档对;(3) aver可扩展至大规模文档集合,而tf-idf仅限成对计算;(4) aver源自简单统计模型下的熵减原理,比tf-idf这种目标导向的构造更具“自然性”。同时我们也指出:(1) aver的计算比tf-idf更复杂;(2) 其尺度无固定单位,使结果解释更困难。

原文摘要 · Abstract (English)

Prior to recent successes using neural networks, term frequency-inverse document frequency (tf-idf) was clearly regarded as the best choice for identifying documents related to a query. We provide a different score, aver, and observe, on a dataset with ground truth marking for association, that aver does do better at finding assciated pairs than tf-idf. This example involves finding associated vertices in a large graph and that may be an area where neural networks are not currently an obvious best choice. Beyond this one anecdote, we observe that (1) aver has a natural threshold for declaring pairs as unassociated while tf-idf does not, (2) aver can distinguish between pairs of documents for which tf-idf gives a score of 1.0, (3) aver can be applied to larger collections of documents than pairs while tf-idf cannot, and (4) that aver is derived from entropy under a simple statistical model while tf-idf is a construction designed to achieve a certain goal and hence aver may be more "natural." To be fair, we also observe that (1) writing down and computing the aver score for a pair is more complex than for tf-idf and (2) that the fact that the aver score is naturally scale-free makes it more complicated to interpret aver scores.

文本匹配熵模型信息检索

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。