提出新框架,高效分割超长文档段落
Toward General Semantic Chunking: A Discriminative Framework for Ultra-Long Documents
- 基于Qwen3-0.6B,用滑动窗口+跨窗融合提升上下文建模能力
- 支持13k token单次输入,推理速度比生成模型快100倍
- 压缩超长段落为单一向量,保持语义且提升检索效率
长文档主题切分在信息检索与文档理解中至关重要,但现有方法在超长文本场景下仍存在明显不足。传统判别模型受固定窗口限制,无法建模文档级语义;生成式大模型虽可输出段落边界,但推理成本高且难以支持长输入。为此,我们提出一种基于Qwen3-0.6B的判别式切分模型,在骨干网络上增加跨窗上下文融合层与边界分类头,并结合重叠滑动窗口策略。模型支持高达13,000令牌的单次输入,可扩展至超长文档的段落边界检测。为进一步提升下游检索效率,我们设计了一种带标量修正的向量融合方法,将超长段落表征压缩为单一向量,无语义损失。在WIKI-727K数据集上的实验表明,相比Jina发布的三个基于Qwen2-0.5B的生成模型,本方法在宏平均F1上表现更优,推理速度提升两个数量级,显著提升长文档处理的实际可用性与可扩展性。
原文摘要 · Abstract (English)
Long-document topic segmentation plays an important role in information retrieval and document understanding, yet existing methods still show clear shortcomings in ultra-long text settings. Traditional discriminative models are constrained by fixed windows and cannot model document-level semantics; generative large language models can output paragraph boundaries, but inference is expensive and long inputs are difficult to support. To address these issues, we propose a discriminative segmentation model based on Qwen3-0.6B. On top of the backbone network, we add a cross-window context fusion layer and a boundary classification head, and combine them with an overlapping sliding-window strategy. Our model supports single-pass inputs of up to 13k tokens and can be extended to ultra-long documents for paragraph boundary detection. To further enhance downstream retrieval efficiency, we derive a vector fusion method with scalar correction, which compresses the representation of ultra-long segments into a single vector without semantic loss. Experiments on the Wikipedia long-document topic segmentation dataset WIKI-727K show that, compared with three generative models based on Qwen2-0.5B released by Jina, our method achieves a better macro-averaged F1 and delivers two orders of magnitude faster inference, substantially improving the practicality and scalability of long-document processing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。