通过精确去重显著减少RAG推理开销，且不影响模型质量。
Byte-Exact Deduplication in Retrieval-Augmented Generation: A Three-Regime Empirical Analysis Across Public Benchmarks
- 采用字节级精确去重，按不同场景分三类评估效果。
- 在对话场景下可压缩80.34%冗余内容，仍保持高质量输出。
- 适用于需要高效推理的工业级AI系统，如企业知识库应用。
本论文对检索增强生成（RAG）流程中的字节级精确块去重进行了实证分析。我们在三种不同运行模式下测量了上下文压缩效果：干净学术检索（2220万贝里数据中减少0.16%字节）、构建的企业模式（减少24.03%）以及多轮对话AI（减少80.34%）。为验证质量保留，我们通过跨厂商五评委校准小组评估四个生产级API（Google Gemini 2.5 Flash、Anthropic Claude Sonnet 4.6、Meta Llama 3.3 70B、OpenAI GPT-5.1）。采用五类人机协作噪声剔除协议处理多数意见差异较大的样本对后，确认字节级去重未引入可测量的质量下降。审计后，所有四家厂商在清洁与高冗余RAG场景下均满足<5%威尔逊95%置信上限的MAT阈值。该工作表明，可在不损害评测级模型质量的前提下，确定性地实现显著推理计算节省。
原文摘要 · Abstract (English)
This preprint presents an empirical analysis of byte-exact chunk-level deduplication in Retrieval-Augmented Generation (RAG) pipelines. We measure context reduction across three distinct operating regimes: clean academic retrieval (0.16% byte reduction on 22.2M BeIR passages), constructed enterprise patterns (24.03% reduction), and multi-turn conversational AI (80.34% reduction). To validate quality preservation, we conducted a cross-vendor 5-judge calibrated panel evaluation across four production APIs (Google Gemini 2.5 Flash, Anthropic Claude Sonnet 4.6, Meta Llama 3.3 70B, and OpenAI GPT-5.1). Applying a five-category human-in-the-loop noise-removal protocol to panel-majority materially different (MAT) pairs, we establish that byte-exact deduplication introduces zero measurable quality regression. Post-audit, all four vendors clear the strict <5% Wilson 95% upper-bound MAT threshold in both the clean and high-redundancy RAG regimes. This work demonstrates that substantial inference compute savings can be achieved deterministically without compromising evaluation-grade model quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。