W-RAC通过智能分块降低RAG系统成本，提升效率与可调试性。
Web Retrieval-Aware Chunking (W-RAC) for Efficient and Cost-Effective Retrieval-Augmented Generation Systems
- 将文本提取与语义分块分离，仅用大模型做分组决策
- 分块成本降低一个数量级，检索效果相当或更优
- 适合大规模网页内容处理，降低幻觉风险
检索增强生成（RAG）系统依赖高效的文档分块策略，在检索质量、延迟和运行成本之间取得平衡。传统分块方法如固定大小、规则驱动或全代理式分块，常因高令牌消耗、冗余文本生成、可扩展性差和可调试性差而受限，尤其在大规模网页内容摄入时更为明显。本文提出一种专为网络文档设计的新型高效分块框架——网络检索感知分块（Web Retrieval-Aware Chunking, W-RAC）。W-RAC通过将解析后的网页内容表示为结构化、可地址的单元，将文本提取与语义分块规划解耦，并仅利用大语言模型（LLM）进行检索感知的分组决策，而非文本生成。该方法显著减少令牌使用，消除幻觉风险，提升系统可观测性。实验分析与架构对比表明，W-RAC在保持或优于传统分块方法的检索性能的同时，将分块相关的LLM成本降低了一个数量级。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) systems critically depend on effective document chunking strategies to balance retrieval quality, latency, and operational cost. Traditional chunking approaches, such as fixed-size, rule-based, or fully agentic chunking, often suffer from high token consumption, redundant text generation, limited scalability, and poor debuggability, especially for large-scale web content ingestion. In this paper, we propose Web Retrieval-Aware Chunking (W-RAC), a novel, cost-efficient chunking framework designed specifically for web-based documents. W-RAC decouples text extraction from semantic chunk planning by representing parsed web content as structured, ID-addressable units and leveraging large language models (LLMs) only for retrieval-aware grouping decisions rather than text generation. This significantly reduces token usage, eliminates hallucination risks, and improves system observability.Experimental analysis and architectural comparison demonstrate that W-RAC achieves comparable or better retrieval performance than traditional chunking approaches while reducing chunking-related LLM costs by an order of magnitude.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。