arXiv:2607.08382cs.IR2026-07

首个针对细粒度文档去重的无监督文本哈希基准,对比多种哈希方法性能。

H3D: Benchmarking Unsupervised Text Hashing for Fine-Grained Document Deduplication

论文配图:H3D: Benchmarking Unsupervised Text Hashing for Fine-Grained Document Deduplication
图 1 · 摘自论文原文
  • 构建统一评估框架,对比传统哈希与语义敏感哈希方法。
  • 在CSFCube和RELISH数据集上验证,发现语义哈希更抗内容改写但开销大。
  • 提供可复现结果,适合文献去重、信息检索研究者参考。

文档哈希能生成紧凑表示以实现高效相似性搜索和文档去重,但现有研究很少在统一协议下对细粒度科学文档的哈希流程进行比较。本文提出H3D,一个面向细粒度文档去重的无监督文本哈希基准。它评估了代表性无监督非学习哈希方法(MinHash、SimHash、Winnowing、FuzzyHash、FlyHash),以及基于冻结BGE嵌入的语义敏感方法,结合两种量化策略(BGE-BIHash和BGE-LSHash)。非学习方法通过手动设计数学规则生成哈希指纹,无需训练或标注相似对,与神经语义哈希模型相区别。我们在两个互补数据集上评估:CSFCube用于细粒度科学文档相似性分析,RELISH用于大规模生物医学相似性搜索。H3D联合报告排名质量(MAP、NDCG@20)、效率和受控文本压缩下的鲁棒性。结果表明存在稳定权衡:词法与结构指纹在近似重复匹配中表现良好,而语义敏感表示在内容改写下更优,但计算成本更高。我们进一步分析不同相似度度量在特定哈希表示下趋于等价的条件,提升方法对比的可解释性和可复现性。

原文摘要 · Abstract (English)

Document hashing provides compact representations for efficient similarity search and document deduplication, but existing studies rarely compare hashing pipelines under a unified protocol for fine-grained scientific documents. H3D is an unsupervised text hashing benchmark for fine-grained document deduplication. It evaluates representative unsupervised non-learning hashing approaches (MinHash, SimHash, Winnowing, FuzzyHash, FlyHash) together with semantic-sensitive methods built from frozen BGE embeddings and two quantization strategies (BGE-BIHash and BGE-LSHash). The non-learning methods generate hash fingerprints through manually designed mathematical rules without training or labeled similarity pairs, which distinguishes them from neural semantic hashing models. We benchmark all methods on CSFCube and RELISH, two datasets that provide complementary evaluation settings: facet-level analysis for scientific-document similarity and larger-scale split-level evaluation for biomedical similarity search. H3D jointly reports ranking quality (MAP, NDCG@20), efficiency, and robustness under controlled text compression. The results show a consistent trade-off: lexical and structural fingerprints are competitive for near-duplicate matching, while semantic-sensitive representations better preserve similarity under content rewriting, at higher computational cost. We further analyze when different similarity measures become rank-equivalent for specific hash representations, improving the interpretability and reproducibility of method comparisons.

文本哈希文档去重无监督学习基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。