提出两种语义感知分块方法，显著提升RAG的检索与生成效果。
Breaking It Down: Domain-Aware Semantic Segmentation for Retrieval Augmented Generation
- 基于投影相似度与度量融合设计新型语义分块算法
- 在PubMedQA上实现24倍MRR提升，且跨域表现优异
- 适合关注RAG系统上下文构建的研究者与工程师
文档分块是检索增强生成（RAG）的关键环节，直接影响相关上下文的检索精度。传统固定长度或递归分块常产生语义不连贯的片段，破坏内容结构。尽管语义分块日益受到关注，其对生成质量的影响仍缺乏深入研究。本文提出两种高效语义分块方法：投影相似度分块（PSC）与度量融合分块（MFC），在PubMed数据上使用三种嵌入模型进行训练。我们构建了评估框架，通过引入完整PubMed Central文章对PubMedQA进行增强，量化分块对检索与生成的影响。结果表明，PSC在MRR上提升24倍，同时在Hits@k指标上表现更优。我们还进行了统计显著性分析及与常见分块库的响应时间对比。尽管仅在单一领域（PubMed）训练，PSC与MFC在多个外部数据集上仍展现出强泛化能力。整体结果证实，特别是PSC，在各项任务中均持续表现卓越。
原文摘要 · Abstract (English)
Document chunking is a crucial component of Retrieval-Augmented Generation (RAG), as it directly affects the retrieval of relevant and precise context. Conventional fixed-length and recursive splitters often produce arbitrary, incoherent segments that fail to preserve semantic structure. Although semantic chunking has gained traction, its influence on generation quality remains underexplored. This paper introduces two efficient semantic chunking methods, Projected Similarity Chunking (PSC) and Metric Fusion Chunking (MFC), trained on PubMed data using three different embedding models. We further present an evaluation framework that measures the effect of chunking on both retrieval and generation by augmenting PubMedQA with full-text PubMed Central articles. Our results show substantial retrieval improvements (24x with PSC) in MRR and higher Hits@k on PubMedQA. We provide a comprehensive analysis, including statistical significance and response-time comparisons with common chunking libraries. Despite being trained on a single domain, PSC and MFC also generalize well, achieving strong out-of-domain generation performance across multiple datasets. Overall, our findings confirm that our semantic chunkers, especially PSC, consistently deliver superior performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。