根据文档特性自动选最优分块方法,提升RAG问答准确率至72%
Adaptive Chunking: Optimizing Chunking-Method Selection for RAG
- 用5个文档内在指标动态选择分块策略
- 准确率从62-64%提升至72%,成功回答问题量增30%以上
- 适合需要高精度检索的法律、技术等场景
RAG的效果高度依赖文档分块方式,即文本被分割为小段以供索引和检索。然而,通用的“一刀切”分块方法常无法捕捉多样化文本的结构与语义特征。尽管分块至关重要,却缺乏独立于下游任务的评估框架。本文提出自适应分块框架,基于五个新提出的文档内在指标——引用完整性(RC)、块内凝聚力(ICC)、文档上下文连贯性(DCC)、块完整性(BI)和大小合规性(SC),为每篇文档选择最合适的分块策略。同时引入两个新分块器:基于LLM的正则表达式分割器和递归分割-合并分块器,并配合针对性后处理。在涵盖法律、技术和社会科学领域的多领域语料上,该方法显著提升下游RAG性能。无需更改模型或提示词,准确率从62%-64%提升至72%,成功回答问题数增加超过30%(65对49)。结果表明,由一组互补的内在指标指导的自适应分块,是构建更鲁棒RAG系统的有效路径。代码已开源。
原文摘要 · Abstract (English)
The effectiveness of Retrieval-Augmented Generation (RAG) is highly dependent on how documents are chunked, that is, segmented into smaller units for indexing and retrieval. Yet, commonly used "one-size-fits-all" approaches often fail to capture the nuanced structure and semantics of diverse texts. Despite its central role, chunking lacks a dedicated evaluation framework, making it difficult to assess and compare strategies independently of downstream performance. We challenge this paradigm by introducing Adaptive Chunking, a framework that selects the most suitable chunking strategy for each document based on a set of five novel intrinsic, document-based metrics: References Completeness (RC), Intrachunk Cohesion (ICC), Document Contextual Coherence (DCC), Block Integrity (BI), and Size Compliance (SC), which directly assess chunking quality across key dimensions. To support this framework, we also introduce two new chunkers, an LLM-regex splitter and a split-then-merge recursive splitter, alongside targeted post-processing techniques. On a diverse corpus spanning legal, technical, and social science domains, our metric-guided adaptive method significantly improves downstream RAG performance. Without changing models or prompts, our framework increases RAG outcomes, raising answers correctness to 72% (from 62-64%) and increasing the number of successfully answered questions by over 30% (65 vs. 49). These results demonstrate that adaptive, document-aware chunking, guided by a complementary suite of intrinsic metrics, offers a practical and effective path to more robust RAG systems. Code available at https://github.com/ekimetrics/adaptive-chunking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。