arXiv:2503.09600cs.CL2025-03ACL被引 32

提出混合分块框架MoC,提升检索增强生成的文本切分质量。

MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented Generation System

  • 设计双指标评估文本分块质量,量化边界清晰度与片段粘性。
  • 引入粒度感知的混合分块器,平衡效率与精度,优于传统方法。
  • 适合需要高质量文本切分的RAG系统开发者使用。

检索增强生成(RAG)虽可补充大语言模型(LLMs),但常忽视管道中的文本分块环节。本文首次提出边界清晰度与片段粘性双指标评估方法,实现分块质量的直接量化。基于此,揭示传统与语义分块在复杂上下文下的局限性,证实将LLM融入分块过程的必要性。为解决基于LLM的分块方法在计算效率与精度间的权衡问题,我们设计了粒度感知的混合分块器(MoC)框架,包含三阶段处理机制。核心目标是引导分块器生成结构化分块正则表达式,进而从原文中提取片段。大量实验表明,所提评估方法与MoC框架有效应对分块挑战,揭示分块关键机制,并提升RAG系统性能。

原文摘要 · Abstract (English)

Retrieval-Augmented Generation (RAG), while serving as a viable complement to large language models (LLMs), often overlooks the crucial aspect of text chunking within its pipeline. This paper initially introduces a dual-metric evaluation method, comprising Boundary Clarity and Chunk Stickiness, to enable the direct quantification of chunking quality. Leveraging this assessment method, we highlight the inherent limitations of traditional and semantic chunking in handling complex contextual nuances, thereby substantiating the necessity of integrating LLMs into chunking process. To address the inherent trade-off between computational efficiency and chunking precision in LLM-based approaches, we devise the granularity-aware Mixture-of-Chunkers (MoC) framework, which consists of a three-stage processing mechanism. Notably, our objective is to guide the chunker towards generating a structured list of chunking regular expressions, which are subsequently employed to extract chunks from the original text. Extensive experiments demonstrate that both our proposed metrics and the MoC framework effectively settle challenges of the chunking task, revealing the chunking kernel while enhancing the performance of the RAG system.

RAG文本分块混合模型评估指标

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。