用预训练网络嵌入块,让长文档检索快1万倍。
REIGN: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling

- 用冻结的引导网络生成分块嵌入,再用对比学习训练双编码器。
- 在专利数据集上性能相当于4.3倍大的模型,参数少得多。
- 适合需要高效长文档检索的工业场景,如专利或法律文本分析。
长文档密集检索成本高昂。基于标记的编码器随序列长度呈二次增长,多数长上下文嵌入模型仅通过架构改造或拉伸百亿参数大模型才能达到32K标记长度。我们提出REIGN(重构嵌入与集成引导网络),一种在冻结引导网络(GN)生成的上下文分块嵌入序列上运行的对比训练双编码器,而非原始标记。REIGN针对多分块输入,主要用于文档到文档检索;单分块输入仍由GN处理。将标记级处理与文档级推理解耦,并将GN嵌入缓存至磁盘,使每文档训练成本相比分块Transformer微调降低约四个数量级。我们还发布了用于长上下文对比训练与评估的合成长文档检索基准。在分布内维基百科基准、分布外LoCo套件及真实世界专利检索案例研究中,REIGN在更小参数预算下表现媲美密集长上下文检索器。配对显著性检验显示,在专利任务上其性能等同于1.6–4.3倍大的模型,且在LoCo上与20倍大的模型差距不超过0.65 nDCG@10。
原文摘要 · Abstract (English)
Dense retrieval over long documents is expensive. Token-level encoders scale quadratically in sequence length, and most long-context embedding models reach 32K tokens only through architectural workarounds or by stretching billion-parameter LLMs. We propose REIGN (Refurbished Embeddings with Integrated Guidance Networks), a contrastively trained bi-encoder that operates on sequences of contextualised chunk embeddings from a frozen Guidance Network (GN) rather than on raw tokens. REIGN targets multi-chunk inputs, primarily for document-to-document retrieval; single-chunk inputs stay with the GN. Decoupling token-level processing from document-level reasoning, and caching the GN embeddings to disk, cuts per-document training cost by roughly four orders of magnitude relative to chunked Transformer fine-tuning. We also release a synthetic long-document retrieval benchmark for contrastive training and evaluation at long context lengths. Across an in-distribution Wikipedia benchmark, the LoCo out-of-distribution suite, and a real-world patent retrieval case study, REIGN matches dense long-context retrievers at smaller parameter budgets in each regime. A paired significance test puts it on par with models 1.6-4.3x larger on the patent task, and it stays within 0.65 nDCG@10 of a 20x-larger model on LoCo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。