跨文档对齐分块提升检索增强生成的准确性
Cross-Document Topic-Aligned Chunking for Retrieval-Augmented Generation
- 在文档集合层面识别主题并合并相关内容为统一片段
- 在HotpotQA上忠实度达0.93，比现有方法高12%
- 适合知识分散在多文档的复杂查询场景
分块质量直接影响RAG系统性能。当前方法独立划分每篇文档，但复杂查询需整合多源信息：存在知识碎片化问题。本文提出跨文档主题对齐（CDTA）分块，在语料库层面重构知识。首先识别文档间主题，将段落映射至对应主题，并合成统一片段。在HotpotQA多跳推理任务中，该方法达成0.93忠实度，优于上下文检索的0.83和语义分块的0.78，较行业最佳实践提升12%（p < 0.05）。在UAE法律文本上，忠实度达0.94，引用准确率0.93。在k=3时，仍保持0.91忠实度，而语义方法降至0.68；单个CDTA片段即可包含传统需多个片段的信息。索引成本较高，但合成后片段信息密度高，降低查询时检索需求。对知识分布于多文档的高并发应用，跨文档合成显著优于单文档优化。
原文摘要 · Abstract (English)
Chunking quality determines RAG system performance. Current methods partition documents individually, but complex queries need information scattered across multiple sources: the knowledge fragmentation problem. We introduce Cross-Document Topic-Aligned (CDTA) chunking, which reconstructs knowledge at the corpus level. It first identifies topics across documents, maps segments to each topic, and synthesizes them into unified chunks. On HotpotQA multi-hop reasoning, our method reached 0.93 faithfulness versus 0.83 for contextual retrieval and 0.78 for semantic chunking, a 12% improvement over current industry best practice (p < 0.05). On UAE Legal texts, it reached 0.94 faithfulness with 0.93 citation accuracy. At k = 3, it maintains 0.91 faithfulness while semantic methods drop to 0.68, with a single CDTA chunk containing information requiring multiple traditional fragments. Indexing costs are higher, but synthesis produces information-dense chunks that reduce query-time retrieval needs. For high-query-volume applications with distributed knowledge, cross-document synthesis improves measurably over within-document optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。