用分层文本分割提升RAG的检索精度
Enhancing Retrieval Augmented Generation with Hierarchical Text Segmentation Chunking
- 分层分割+聚类生成语义更连贯的文本块
- 在NarrativeQA等3个数据集上效果更好
- 适合需要精准知识检索的应用场景
检索增强生成（RAG）系统通常采用分块策略进行检索，通过让大语言模型访问外部知识，确保信息更新且领域特定。然而，传统方法常因忽略文本内在结构，导致分块无法充分捕捉语义。本文提出一种新框架，结合分层文本分割与聚类，生成更具语义一致性的文本块。推理时，该框架利用段落级和簇级向量表示进行检索，提高获取精确、上下文相关信息的可能性。在NarrativeQA、QuALITY和QASPER数据集上的评估显示，该方法优于传统分块技术。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) systems commonly use chunking strategies for retrieval, which enhance large language models (LLMs) by enabling them to access external knowledge, ensuring that the retrieved information is up-to-date and domain-specific. However, traditional methods often fail to create chunks that capture sufficient semantic meaning, as they do not account for the underlying textual structure. This paper proposes a novel framework that enhances RAG by integrating hierarchical text segmentation and clustering to generate more meaningful and semantically coherent chunks. During inference, the framework retrieves information by leveraging both segment-level and cluster-level vector representations, thereby increasing the likelihood of retrieving more precise and contextually relevant information. Evaluations on the NarrativeQA, QuALITY, and QASPER datasets indicate that the proposed method achieved improved results compared to traditional chunking techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。