Merlin通过字节级去重加速大模型推理，提升效率且不损失数据
Merlin: Deterministic Byte-Exact Deduplication for Lossless Context Optimization in Large Language Model Inference
- 用优化的哈希表和快速算法实现字节精确去重
- 在高冗余数据中减少超71%输入量，保持数据完全一致
- 适合需要高效文本处理的大模型应用，如RAG系统
从大规模检索系统到复杂数据流水线，数据密集型应用正受高度冗余文本语料处理瓶颈制约。我们提出Merlin，一种本地优先、无特定依赖、高吞吐的去重与上下文优化引擎，旨在缓解此类低效问题。结合高度优化的SIMD友好开放寻址扁平哈希集合与xxHash3-64，Merlin可快速实现文本段落与数据块的字节级精确去重。该技术广泛适用于各类文本处理流程，在大语言模型（LLM）生态中尤为显著，如检索增强生成（RAG）。实证评估显示，其在低冗余数据集上实现13.9%的输入缩减，在高冗余流水线中超过71%；同时保证绝对数据保真度。此外，本文详述通过模型上下文协议（MCP）构建的集成架构，支持在主流IDE和自主代理中安全、零网络介入部署。论文还阐述了核心算法设计、性能基准及以8.7 GB/s持续速度处理数据所需架构原则。
原文摘要 · Abstract (English)
Data-intensive applications, ranging from large-scale retrieval systems to advanced data pipelines, are increasingly bottlenecked by the processing of highly redundant text corpora. We present Merlin, a local-first, agnostic, high-throughput deduplication and context optimization engine designed to mitigate these inefficiencies. Utilizing a highly optimized, SIMD-friendly open-addressing flat hash set combined with xxHash3-64, Merlin performs rapid, byte-exact deduplication of text passages and data chunks. While broadly applicable to any text-processing workflow, its impact is particularly pronounced in Large Language Model (LLM) ecosystems, such as Retrieval-Augmented Generation (RAG). Our empirical evaluations demonstrate an input reduction ranging from 13.9% in low-redundancy datasets to over 71% in high-redundancy pipelines, maintaining absolute data fidelity. Furthermore, we detail the system's integration architecture via the Model Context Protocol (MCP), enabling secure, zero-network-interception deployment across major IDEs and autonomous agents. This paper outlines the core algorithmic design, performance benchmarks, and the architectural principles required to process data at sustained speeds of up to 8.7 GB/s.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。