arXiv:2602.14784cs.IRcs.AI2026-02

根据用户可能提问的意图,动态切分文档以提升检索准确率。

Intent-Driven Dynamic Chunking: Segmenting Documents to Reflect Predicted Information Needs

  • 用大模型预测用户意图,指导文档切分边界。
  • 在6个数据集上提升检索准确率5%至67%,减少40%-60%碎片数。
  • 适合长文档、异构文本的检索增强生成系统使用。

将长文档切分为小段是信息检索中的基础挑战。无论用于搜索引擎、问答系统或检索增强生成(RAG),有效的分割直接影响系统定位与返回相关资讯的能力。然而,传统方法如固定长度或基于连贯性的分割忽略用户意图,导致答案被切碎或包含无关噪声。本文提出意图驱动的动态切分(IDC),利用大语言模型生成文档的潜在用户意图,并通过动态规划算法寻找全局最优的切分边界。这一方法首次将动态规划应用于意图感知的切分,避免了贪心策略的局限。我们在六个多样化问答数据集上评估了 IDC,涵盖新闻、维基百科、学术论文和技术文档。IDC 在五个数据集上优于传统切分策略,顶1检索准确率提升5%至67%,第六个数据集表现与最佳基线相当。同时,IDC 产生的片段数量减少40%-60%,但答案覆盖率保持93%-100%。结果表明,将文档结构与预期信息需求对齐,显著提升了长而异质文档的检索性能。

原文摘要 · Abstract (English)

Breaking long documents into smaller segments is a fundamental challenge in information retrieval. Whether for search engines, question-answering systems, or retrieval-augmented generation (RAG), effective segmentation determines how well systems can locate and return relevant information. However, traditional methods, such as fixed-length or coherence-based segmentation, ignore user intent, leading to chunks that split answers or contain irrelevant noise. We introduce Intent-Driven Dynamic Chunking (IDC), a novel approach that uses predicted user queries to guide document segmentation. IDC leverages a Large Language Model to generate likely user intents for a document and then employs a dynamic programming algorithm to find the globally optimal chunk boundaries. This represents a novel application of DP to intent-aware segmentation that avoids greedy pitfalls. We evaluated IDC on six diverse question-answering datasets, including news articles, Wikipedia, academic papers, and technical documentation. IDC outperformed traditional chunking strategies on five datasets, improving top-1 retrieval accuracy by 5% to 67%, and matched the best baseline on the sixth. Additionally, IDC produced 40-60% fewer chunks than baseline methods while achieving 93-100% answer coverage. These results demonstrate that aligning document structure with anticipated information needs significantly boosts retrieval performance, particularly for long and heterogeneous documents.

文档切分RAG意图预测动态规划

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。