改进文档分块方法，提升问答系统准确率
Passage Segmentation of Documents for Extractive Question Answering
- 根据上下文语义动态划分不同粒度的文本块
- 在两个基准数据集上显著提升检索效果
- 适合需要高质量文档处理的问答系统研发者
检索增强生成（RAG）在开放域问答中表现优异，但其关键的分块过程常被忽视。本研究强调分块对密集段落检索和端到端RAG流程的重要影响。提出一种新型日志引导多粒度分块框架（LGMGC），可将长文档拆分为具有上下文意义且自包含的多粒度块。在两个基准数据集上的实验表明，LGMGC不仅优化了检索阶段性能，集成到RAG流程后也优于现有分块方法。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) has proven effective in open-domain question answering. However, the chunking process, which is essential to this pipeline, often receives insufficient attention relative to retrieval and synthesis components. This study emphasizes the critical role of chunking in improving the performance of both dense passage retrieval and the end-to-end RAG pipeline. We then introduce the Logits-Guided Multi-Granular Chunker (LGMGC), a novel framework that splits long documents into contextualized, self-contained chunks of varied granularity. Our experimental results, evaluated on two benchmark datasets, demonstrate that LGMGC not only improves the retrieval step but also outperforms existing chunking methods when integrated into a RAG pipeline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。