用大模型+高精度编码实现文本无损压缩新纪录
Nacrith: Neural Lossless Compression via Ensemble Context Modeling and High-Precision CDF Coding
- 结合大模型与轻量预测器，用32位算术编码提升精度
- 在alice29上达0.918 bpb，比gzip快3.1倍，突破香农熵极限
- 支持任意二进制文件压缩，可在消费级显卡运行
我们提出Nacrith，一种结合135M参数Transformer语言模型（SmolLM2-135M）与轻量在线预测器及32位算术编码的无损压缩系统，在自然语言文本压缩中取得当前最佳效果。相比基础大模型加算术编码范式，Nacrith引入多项创新：(1) 将累积分布函数（CDF）精度从2^16提升至2^24，消除大规模词表中约75%的最小概率地板导致的量化开销；(2) 基于词元级N-gram模型实现快速局部预测；(3) 采用自适应对数空间偏置头，通过在线梯度下降校正文档级大模型误差；(4) 根据置信度跳过高度可预测词元以加速；(5) 提出混合二进制格式NC06，首次实现基于大模型的任意二进制文件压缩；(6) 采用llama.cpp推理后端，单标记解码速度较PyTorch快约7倍；(7) 支持最多8个工作节点的并行多GPU压缩；(8) 原生KV缓存滑动窗口机制，使每滑动开销降低约37倍。系统仅需约500 MB GGUF权重和每工作节点1.2 GB VRAM，可在消费级GPU运行。在alice29（Canterbury Corpus，152 KB）上，达到0.918 bits per byte（bpb），优于gzip（3.1x）、bzip2（2.5x）、CMIX v21（44%）和ts_zip（20%），且压缩率低于0阶、1阶、2阶字节级香农熵边界。在enwik8（100 MB）上达0.9389 bpb（11.74%），优于ts_zip（~1.11 bpb）15%、FineZip（1.024 bpb）8%，尽管使用60倍更小的模型且未微调。对训练截止后发布的文档进行域外评估，结果为0.723 bpb，证明性能非记忆效应所致。
原文摘要 · Abstract (English)
We present Nacrith, a lossless compression system that combines a 135M-parameter transformer language model (SmolLM2-135M) with an ensemble of lightweight online predictors and a 32-bit arithmetic coder, achieving the best compression results among the systems evaluated in this study on natural language text. Beyond the base LLM-plus-arithmetic-coding paradigm, Nacrith introduces several contributions: (1) a CDF precision upgrade from 2^16 to 2^24 that eliminates ~75% of quantization overhead caused by minimum-probability floors in large vocabularies; (2) a token-level N-gram model for fast local predictions; (3) an adaptive log-space bias head correcting per-document LLM errors via online gradient descent; (4) confidence-based LLM skip for accelerating highly predictable tokens; (5) a hybrid binary format (NC06) extending neural compression to arbitrary binary files--to our knowledge a first among LLM-based compressors; (6) a llama cpp inference backend achieving ~7x faster single-token decode than PyTorch; (7) parallel multi-GPU compression across up to 8 workers; and (8) native KV cache sliding window reducing per-slide cost by ~37x. The system requires only ~500 MB of GGUF weights and ~1.2 GB VRAM per worker, running on consumer GPUs. On alice29 (Canterbury Corpus, 152 KB), Nacrith achieves 0.918 bits per byte (bpb)--outperforming gzip by 3.1x, bzip2 by 2.5x, CMIX v21 by 44%, and ts_zip by 20%, while compressing below the 0th-, 1st-, and 2nd-order byte-level Shannon entropy bounds. On enwik8 (100 MB), Nacrith achieves 0.9389 bpb (11.74%), surpassing ts_zip (~1.11 bpb) by 15% and FineZip (1.024 bpb) by 8% despite using a 60x smaller model with no fine-tuning. An out-of-distribution (OOD) evaluation on a document published after the model's training cutoff confirms these gains are not memorization artifacts, achieving 0.723 bpb on unseen text.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。