结合布局与语义分析，提升复杂文档分块准确性
S2 Chunking: A Hybrid Framework for Document Segmentation Through Integrated Spatial and Semantic Analysis
- 融合边界框与文本嵌入构建加权图，用谱聚类分块
- 在多栏、报告类文档中表现优于传统方法，准确率显著提升
- 支持自定义最大令牌数，适合输入受限的模型应用
文档分块是自然语言处理中的关键任务，涉及将文档划分为有意义的段落。传统方法仅依赖语义分析，忽视元素的空间布局，而布局关系对理解复杂文档至关重要。本文提出一种混合方法，整合版面结构、语义分析和空间关系，以增强分块的一致性与准确性。通过利用边界框（bbox）信息与文本嵌入，构建文档元素的加权图表示，并采用谱聚类进行划分。实验结果表明，该方法在具有多样化布局的文档（如报告、文章、多栏设计）中表现更优。此外，该方法可确保每个分块不超过指定的令牌长度，适用于输入大小受限的语言模型场景。
原文摘要 · Abstract (English)
Document chunking is a critical task in natural language processing (NLP) that involves dividing a document into meaningful segments. Traditional methods often rely solely on semantic analysis, ignoring the spatial layout of elements, which is crucial for understanding relationships in complex documents. This paper introduces a novel hybrid approach that combines layout structure, semantic analysis, and spatial relationships to enhance the cohesion and accuracy of document chunks. By leveraging bounding box information (bbox) and text embeddings, our method constructs a weighted graph representation of document elements, which is then clustered using spectral clustering. Experimental results demonstrate that this approach outperforms traditional methods, particularly in documents with diverse layouts such as reports, articles, and multi-column designs. The proposed method also ensures that no chunk exceeds a specified token length, making it suitable for use cases where token limits are critical (e.g., language models with input size limitations)
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。