arXiv:2602.16974cs.IR2026-02被引 3

系统对比文档切分策略,发现不同任务下最优方法不同。

Beyond Chunk-Then-Embed: A Comprehensive Taxonomy and Evaluation of Document Chunking Strategies for Information Retrieval

  • 按分割方式和嵌入时机构建切分策略分类框架
  • 简单结构化切分在整体检索中表现更好,而LLM引导切分适合精准定位
  • 揭示切分策略效果与任务类型强相关,非仅由块大小决定

文档切分是稠密检索系统中的关键预处理步骤,但其设计空间仍不清晰。近期研究提出了多种并行方法,包括基于大模型的(如DenseX、LumberChunker)和上下文感知策略(如Late Chunking),这些方法在分段前生成嵌入以保留语义信息。然而,这些方法独立发展且评估基准重叠少,难以直接比较。本文复现了先前研究,提出一个系统性框架,从两个维度统一现有策略:(1) 分割方法,包括基于结构的(固定大小、句子级、段落级)以及语义驱动和LLM引导方法;(2) 嵌入范式,决定切分与嵌入的先后顺序(预嵌入切分 vs. 上下文感知切分)。在两种典型检索场景中进行评估:文档内检索(针堆找针)和文档集合检索(标准信息检索任务)。结果表明,最优切分策略依赖于任务类型:简单结构化方法在文档集合检索中表现更优,而LumberChunker在文档内检索中最佳;上下文感知切分提升集合检索效果,但降低文档内检索性能。此外,块大小与文档内检索效果呈中度相关,与集合检索弱相关,说明分割方法差异并非仅由块大小驱动。代码与评估基准已公开。

原文摘要 · Abstract (English)

Document chunking is a critical preprocessing step in dense retrieval systems, yet the design space of chunking strategies remains poorly understood. Recent research has proposed several concurrent approaches, including LLM-guided methods (e.g., DenseX and LumberChunker) and contextualized strategies(e.g., Late Chunking), which generate embeddings before segmentation to preserve contextual information. However, these methods emerged independently and were evaluated on benchmarks with minimal overlap, making direct comparisons difficult. This paper reproduces prior studies in document chunking and presents a systematic framework that unifies existing strategies along two key dimensions: (1) segmentation methods, including structure-based methods (fixed-size, sentence-based, and paragraph-based) as well as semantically-informed and LLM-guided methods; and (2) embedding paradigms, which determine the timing of chunking relative to embedding (pre-embedding chunking vs. contextualized chunking). Our reproduction evaluates these approaches in two distinct retrieval settings established in previous work: in-document retrieval (needle-in-a-haystack) and in-corpus retrieval (the standard information retrieval task). Our comprehensive evaluation reveals that optimal chunking strategies are task-dependent: simple structure-based methods outperform LLM-guided alternatives for in-corpus retrieval, while LumberChunker performs best for in-document retrieval. Contextualized chunking improves in-corpus effectiveness but degrades in-document retrieval. We also find that chunk size correlates moderately with in-document but weakly with in-corpus effectiveness, suggesting segmentation method differences are not purely driven by chunk size. Our code and evaluation benchmarks are publicly available at (Anonymoused).

信息检索文档切分稠密检索LLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。