用长文本嵌入模型先整体编码再切块，提升检索效果。
Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models
- 先整体编码再分块，保留上下文信息
- 在多个检索任务中表现更优
- 无需额外训练，适配多种模型
许多应用场景需要检索文本的较小片段，而密集向量检索系统在处理较短文本段时表现更好，因为语义不易被压缩。因此，实践者通常将长文档拆分为小段分别编码。然而，这种分块方式会丢失相邻段落的上下文信息，导致表示不充分。本文提出一种新方法——晚切块（late chunking），利用长上下文嵌入模型先对整个长文本的所有标记进行编码，再在Transformer模型之后、平均池化之前进行分块。由此产生的块嵌入能捕捉完整上下文信息，在多种检索任务中取得更优结果。该方法通用性强，可应用于多种长上下文嵌入模型，且无需额外训练。为进一步提升效果，我们还提出了针对嵌入模型的专用微调方法。
原文摘要 · Abstract (English)
Many use cases require retrieving smaller portions of text, and dense vector-based retrieval systems often perform better with shorter text segments, as the semantics are less likely to be over-compressed in the embeddings. Consequently, practitioners often split text documents into smaller chunks and encode them separately. However, chunk embeddings created in this way can lose contextual information from surrounding chunks, resulting in sub-optimal representations. In this paper, we introduce a novel method called late chunking, which leverages long context embedding models to first embed all tokens of the long text, with chunking applied after the transformer model and just before mean pooling - hence the term late in its naming. The resulting chunk embeddings capture the full contextual information, leading to superior results across various retrieval tasks. The method is generic enough to be applied to a wide range of long-context embedding models and works without additional training. To further increase the effectiveness of late chunking, we propose a dedicated fine-tuning approach for embedding models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。