arXiv:2509.15658cs.IRcs.AI2025-09

将文档分块生成标题、问题和关键词,提升检索精度与效率。

Chunk Knowledge Generation Model for Enhanced Information Retrieval: A Multi-task Learning Approach

  • 分块处理文档,用T5模型并行生成标题、问题和关键词。
  • 在Top@10上达到95.41%准确率,优于传统分块检索。
  • 适合大规模信息检索系统,尤其关注效率与内容可靠性。

传统查询扩展方法易受上下文影响,性能不稳定;文档扩展虽受关注,但如Doc2Query存在预处理成本高、索引膨胀及生成内容不可靠等问题。为此,本文提出一种分块知识生成模型,将文档切分为块,针对每一块生成文本信息以提升检索效率与准确率。该模型基于T5的多任务学习结构,通过一次编码、两次解码,同步生成标题、候选问题,并从用户查询中提取关键词,实现三类语义信息的并行处理。生成内容作为额外信息融入检索系统。基于GPT对305组查询-文档对的评估显示,使用该模型的检索在Top@10上达到95.41%准确率,显著优于仅基于文档分块的检索方法。本研究贡献在于提出一种在检索流水线中同时生成标题与候选问题的分块生成方案,并通过定性评估为大规模信息检索系统提供了可落地的性能提升证据。

原文摘要 · Abstract (English)

Traditional query expansion techniques for addressing vocabulary mismatch problems in information retrieval are context-sensitive and may lead to performance degradation. As an alternative, document expansion research has gained attention, but existing methods such as Doc2Query have limitations including excessive preprocessing costs, increased index size, and reliability issues with generated content. To mitigate these problems and seek more structured and efficient alternatives, this study proposes a method that divides documents into chunk units and generates textual data for each chunk to simultaneously improve retrieval efficiency and accuracy. The proposed "Chunk Knowledge Generation Model" adopts a T5-based multi-task learning structure that simultaneously generates titles and candidate questions from each document chunk while extracting keywords from user queries. This approach maximizes computational efficiency by generating and extracting three types of semantic information in parallel through a single encoding and two decoding processes. The generated data is utilized as additional information in the retrieval system. GPT-based evaluation on 305 query-document pairs showed that retrieval using the proposed model achieved 95.41% accuracy at Top@10, demonstrating superior performance compared to document chunk-level retrieval. This study contributes by proposing an approach that simultaneously generates titles and candidate questions from document chunks for application in retrieval pipelines, and provides empirical evidence applicable to large-scale information retrieval systems by demonstrating improved retrieval accuracy through qualitative evaluation.

信息检索多任务学习文本生成分块处理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。