不依赖模型的高效文档解析法,速度快能耗低。
ChunkNorris: A High-Performance and Low-Energy Approach to PDF Parsing and Chunking
- 用简单启发式规则替代机器学习,实现高效解析
- 实测速度更快、耗能更低,检索准确率仍领先
- 适合资源受限场景,开源数据集可复现
在检索增强生成(RAG)应用中,信息检索环节至关重要,它为大语言模型提供上下文以生成恰当且真实响应。高质量的文档解析与分块直接影响下游任务表现。本文提出ChunkNorris,一种基于启发式规则的PDF解析与分块新方法,不依赖机器学习,通过一系列简单但高效的规则,在极低计算开销下实现高性能。我们通过全面基准测试验证其效果,对比了执行时间、能耗及检索准确率等指标。实验表明,ChunkNorris优于基线和更复杂的方法,在多个指标上表现突出。研究还公开了一个数据集以支持结果复现。该工作凸显了启发式方法在实际、资源受限的RAG场景中的潜力。
原文摘要 · Abstract (English)
In Retrieval-Augmented Generation applications, the Information Retrieval part is central as it provides the contextual information that enables a Large Language Model to generate an appropriate and truthful response. High quality parsing and chunking are critical as efficient data segmentation directly impacts downstream tasks, i.e. Information Retrieval and answer generation. In this paper, we introduce ChunkNorris, a novel heuristic-based technique designed to optimise the parsing and chunking of PDF documents. Our approach does not rely on machine learning and employs a suite of simple yet effective heuristics to achieve high performance with minimal computational overhead. We demonstrate the efficiency of ChunkNorris through a comprehensive benchmark against existing parsing and chunking methods, evaluating criteria such as execution time, energy consumption, and retrieval accuracy. We propose an open-access dataset to produce our results. ChunkNorris outperforms baseline and more advanced techniques, offering a practical and efficient alternative for Information Retrieval tasks. Therefore, this research highlights the potential of heuristic-based methods for real-world, resource-constrained RAG use cases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。