用无训练的高维哈希突破稠密检索瓶颈，首次超越传统BM25。
Beyond the Geometric Curse: High-Dimensional N-Gram Hashing for Dense Retrieval
- 用字符哈希直接映射文本到高维空间，无需训练。
- 在32,768维下达到93.90% Recall@100，超过BM25的93.6%。
- 适合追求高效、无训练部署的检索系统应用。
为何即使强大的70亿参数嵌入模型也难以应对简单的检索任务，而几十年前的BM25却能轻松处理？近期理论表明，这是由于维度瓶颈所致：将无限的语言细微差别压缩进固定长度的向量中。我们提出NUMEN，通过完全去除学习过程来打破这一瓶颈。不同于训练复杂层将文本映射到受限空间，NUMEN采用确定性的字符哈希，直接将语言投影到高维向量。该方法无需训练，支持无限词汇量，且几何容量可按需扩展。在LIMIT基准上，NUMEN在32,768维下实现93.90% Recall@100，是首个正式超越稀疏检索基线BM25（93.6%）的稠密检索模型。研究发现，稠密检索真正的瓶颈不在架构，而在嵌入层本身。解决方案不一定是更智能的训练，而是给予更多表达空间。
原文摘要 · Abstract (English)
Why do even the most powerful 7B-parameter embedding models struggle with simple retrieval tasks that the decades old BM25 handles with ease? Recent theory suggests that this happens because of a dimensionality bottleneck. This occurs when we force infinite linguistic nuances into small, fixed-length learned vectors. We developed NUMEN to break this bottleneck by removing the learning process entirely. Instead of training heavy layers to map text to a constrained space, NUMEN uses deterministic character hashing to project language directly onto high-dimensional vectors. This approach requires no training, supports an unlimited vocabulary, and allows the geometric capacity scale as needed. On the LIMIT benchmark, NUMEN achieves 93.90 % Recall@100 at 32,768 dimensions. This makes it the first dense retrieval model to officially surpass the sparse BM25 baseline 93.6 %. Our findings show that the real problem in dense retrieval isn't the architecture, but the embedding layer itself. The solution isn't necessarily smarter training, but simply providing more room to breathe.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。