用自限梯度更新压缩语言模型嵌入，提升检索效率与效果。
DIVE: Embedding Compression via Self-Limiting Gradient Updates
- 设计自限铰链损失，仅更新未满足的排序约束。
- 在5个BEIR基准上优于6种基线方法，256维压缩后仍领先。
- 适合需要高效存储与推理的检索系统部署。
高维语言模型嵌入增加存储与搜索开销，而监督压缩器在相关性标签稀缺时易过拟合。我们提出DIVE（隐式视图集成的降维），一种与自限铰链损失、几何蒸馏及头级NT-Xent联合设计的残差压缩适配器。铰链损失停止已满足排序约束的更新，密集目标稳定压缩表示；推理时仅保留首个头部。在两个LLM2Vec主干、五个BEIR基准、128维和256维输出、六种基线条件下，DIVE在全部五个主要基准上表现最佳，同时优于PCA和自编码器等无监督压缩方法。
原文摘要 · Abstract (English)
High-dimensional language-model embeddings increase storage and search costs, while supervised compressors can overfit when relevance labels are scarce. We present DIVE (Dimensionality reduction with Implicit View Ensembles), a residual compression adapter codesigned with a self-limiting hinge loss, geometry distillation, and head-wise NT-Xent over implicit coordinate views. The hinge stops updating satisfied ranking constraints, while the dense objectives stabilize the compressed representation; only the first head is retained at inference. Under query-disjoint evaluation with two LLM2Vec backbones, five BEIR benchmarks, 128d and 256d outputs, and six baselines, DIVE is the strongest adapter on all five primary benchmarks. It also outperforms PCA and an autoencoder in comparisons against unsupervised compressors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。