用多粒度语义哈希高效去重，大幅降低验证开销。
SemHash-LLM: A Multi-Granularity Semantic Hashing Framework for Document Deduplication
- 融合字符、词元和文档级信号，分层筛选候选文档。
- 仅需不到1%的神经网络验证成本，保持高精度去重。
- 适合海量文本去重场景，尤其抗模板污染和短文本干扰。
大规模文档去重需在保留语义等价性的同时保证效率。我们提出 SemHash LLM，一种多粒度框架，统一了语义投影哈希、注意力加权 MinHash、对比边界学习与选择性 LLM 判定。通过门控融合字符、词元和文档级信号，再经级联过滤管道实现高效候选缩减。语义投影哈希在蒸馏 LLM 嵌入空间中学习紧凑二进制码，注意力加权 MinHash 抑制冗余内容并突出关键信息。自适应决策边界与不确定性估计增强了对模板污染、短文本扰动、包含关系及病毒片段的鲁棒性。实验表明，SemHash LLM 在低于 1% 的神经验证成本下实现优异的去重效果。
原文摘要 · Abstract (English)
Large scale document deduplication must preserve semantic equivalence while remaining efficient over massive corpora. We present SemHash LLM, a multi granularity framework that unifies semantic projection hashing, attention weighted MinHash, contrastive boundary learning, and selective LLM based adjudication. The method combines character, token, and document level signals through gated fusion, then applies a cascaded filtering pipeline for efficient candidate reduction. Semantic projection hashing learns compact binary codes in distilled LLM embedding space, while attention weighted Min- Hash suppresses boilerplate and emphasizes informative content. Adaptive decision boundaries and uncertainty estimation further improve robustness across template pollution, short text perturbation, containment, and viral fragments. Experiments show that SemHash LLM achieves strong duplicate detection quality with less than one percent neural verification cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。