不同文档类型需配不同分块大小，小块适合事实型，大块利于上下文理解。
Rethinking Chunk Size For Long-Document Retrieval: A Multi-Dataset Analysis
- 按64-128或512-1024令牌分块，根据数据特点选择最优策略。
- 小块提升事实型数据检索准确率，大块增强长上下文理解能力。
- 模型敏感度差异大，Stella适配大块，Snowflake偏好小块。
分块是检索增强生成（RAG）系统中的关键预处理步骤，显著影响跨多种数据集的检索效果。本研究系统评估了固定大小分块策略及其对检索性能的影响，使用多种嵌入模型在短文本和长文本数据集上进行实验。结果表明，分块大小对检索效果有决定性影响：64-128令牌的小块适用于简洁、以事实为主的答案数据集，而512-1024令牌的大块更有利于需要广泛上下文理解的数据集。我们还分析了分块对不同嵌入模型的影响，发现模型对分块策略敏感度各异——Stella模型因能利用全局上下文，在大块下表现更优；Snowflake则在小块下更具优势，擅长细粒度实体匹配。研究揭示了分块大小、嵌入模型与数据特征之间的权衡关系，强调需改进分块质量评估指标，并构建更全面的数据集，以推动长文档信息检索中基于分块的检索技术发展。
原文摘要 · Abstract (English)
Chunking is a crucial preprocessing step in retrieval-augmented generation (RAG) systems, significantly impacting retrieval effectiveness across diverse datasets. In this study, we systematically evaluate fixed-size chunking strategies and their influence on retrieval performance using multiple embedding models. Our experiments, conducted on both short-form and long-form datasets, reveal that chunk size plays a critical role in retrieval effectiveness -- smaller chunks (64-128 tokens) are optimal for datasets with concise, fact-based answers, whereas larger chunks (512-1024 tokens) improve retrieval in datasets requiring broader contextual understanding. We also analyze the impact of chunking on different embedding models, finding that they exhibit distinct chunking sensitivities. While models like Stella benefit from larger chunks, leveraging global context for long-range retrieval, Snowflake performs better with smaller chunks, excelling at fine-grained, entity-based matching. Our results underscore the trade-offs between chunk size, embedding models, and dataset characteristics, emphasizing the need for improved chunk quality measures, and more comprehensive datasets to advance chunk-based retrieval in long-document Information Retrieval (IR).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。