用极低比特量化让大模型文本嵌入更省资源，兼顾速度与存储。
BitNet Text Embeddings
- 将大模型转为三值权重+量化激活的轻量架构，提升编码效率。
- 在MMTEB上性能接近全精度模型，支持多种存储精度灵活切换。
- 适合需要低延迟、低存储的实时检索系统部署。
基于大语言模型的文本嵌入显著提升了检索与语义表示质量，但部署成本高：大模型推理慢，高维全精度嵌入带来巨大存储与带宽压力。本文提出BITEMBED，一种面向LLM文本嵌入的极端低比特框架，同时优化编码效率与向量存储。BITEMBED将预训练LLM主干转换为类似BitNet的嵌入编码器，采用三值权重、量化激活和轻量归一化精修。模型通过持续对比预训练适应表征学习，并经由监督对比微调，融合来自全精度教师模型的相似性分布蒸馏与注意力关系蒸馏。除主干量化外，BITEMBED还训练输出嵌入以支持多种存储精度，满足不同场景需求。在MMTEB（eng, v2）上使用Qwen3-0.6B和Gemma3-270M的实验表明，BITEMBED性能接近全精度教师模型。此外，其可灵活生成不同精度的文本嵌入，在性能与存储成本间实现良好权衡。
原文摘要 · Abstract (English)
LLM-based text embedders have substantially improved retrieval and semantic representation quality, but their deployment remains costly: large backbone models slow down embedding inference, while high-dimensional full-precision embeddings impose substantial storage and bandwidth overhead on large-scale indexes. In this paper, we present BITEMBED, an extreme low-bit framework for LLM-based text embedding that jointly targets encoding efficiency and vector storage. BITEMBED converts pretrained LLM backbones into BitNet-style embedding encoders with ternary weights, quantized activations, and lightweight normalization refinement. The converted model is adapted to representation learning through continual contrastive pre-training, followed by supervised contrastive fine-tuning with both similarity-distribution distillation and attention-relation distillation from a full-precision teacher. Beyond quantizing the backbone, BITEMBED further trains output embeddings to support multiple storage precisions meeting different storage needs in various scenarios. Experiments on MMTEB (eng, v2) with Qwen3-0.6B and Gemma3-270M show that BITEMBED is largely comparable to full precision teacher embedders. Moreover, BITEMBED flexibly obtains text embeddings of various precisions, achieving a trade-off between performance and storage cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。