超长文本嵌入模型，支持128K tokens，提升检索生成效果
Dewey Long Context Embedding Model: A Technical Report
- 通过分块对齐训练，同时生成局部和全局嵌入
- 在MTEB和LongEmbed基准上表现优异，支持128K上下文
- 适合需要长文档理解的检索增强生成场景
本技术报告介绍了开源模型dewey_en_beta的训练方法与评估结果。随着检索增强生成（RAG）系统的发展及大语言模型（LLM）上下文窗口的扩展，传统嵌入模型面临严峻挑战：处理超过典型序列长度的文档时，语义连贯性难以维持，严重影响知识密集型应用的检索性能。本文提出的dewey_en_beta模型，在MTEB（Eng, v2）和LongEmbed基准上表现优异，支持128K token序列。其核心技术为分块对齐训练，通过知识蒸馏实现局部分块嵌入与全局文档表示的同步生成。模型发布信息见https://huggingface.co/infgrad/dewey_en_beta。
原文摘要 · Abstract (English)
This technical report presents the training methodology and evaluation results of the open-source dewey_en_beta embedding model. The increasing demand for retrieval-augmented generation (RAG) systems and the expanding context window capabilities of large language models (LLMs) have created critical challenges for conventional embedding models. Current approaches often struggle to maintain semantic coherence when processing documents exceeding typical sequence length limitations, significantly impacting retrieval performance in knowledge-intensive applications. This paper presents dewey_en_beta, a novel text embedding model that achieves excellent performance on MTEB (Eng, v2) and LongEmbed benchmark while supporting 128K token sequences. Our technical contribution centers on chunk alignment training, an innovative methodology that enables the simultaneous generation of localized chunk embeddings and global document-level representations through distillation. Information regarding the model release can be found at https://huggingface.co/infgrad/dewey_en_beta.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。