用自研模型和稀疏词组记忆,实现无需预训练的高效无损压缩。
StateSMix: Online Lossless Compression via Mamba State Space Models and Sparse N-gram Context Mixing
- 自训练Mamba结构结合稀疏词组哈希,实时更新概率预测
- 在enwik8上达2.123~2.162 bpb,优于传统压缩工具8.7%
- 纯C实现,支持多核加速,每秒处理约2000个词元
我们提出StateSMix,一种完全自包含的无损压缩器,将在线训练的Mamba型状态空间模型(SSM)与稀疏n-gram上下文混合及算术编码结合。模型从零初始化,逐标记在待压缩文件上训练,无需预训练权重、无需GPU、无外部依赖。SSM(DM=32, NL=2,每文件约12万活跃参数)提供BPE词元的概率连续估计;九个稀疏n-gram哈希表(从二元到32元,每表1600万槽位)通过软最大不变的对数偏置机制,仅更新非零计数项,实现精确的局部与长程模式记忆。熵自适应缩放机制根据SSM预测置信度调节n-gram贡献,防止神经模型已校准时的过矫正。在标准enwik8基准上,StateSMix在1MB、3MB、10MB数据上分别达到2.123、2.149、2.162 bpb,较xz -9e(LZMA2)提升8.7%、5.4%、0.7%。消融实验表明:仅使用SSM即比频率统计基线减少46.6%体积,且超越xz无n-gram组件;而n-gram表通过精确上下文记忆带来额外4.1%增益。OpenMP并行化训练循环在4核下提速1.9倍。系统纯用C实现,支持AVX2 SIMD,可在通用x86-64硬件上以约2000词元/秒速度处理。
原文摘要 · Abstract (English)
We present StateSMix, a fully self-contained lossless compressor that couples an online-trained Mamba-style State Space Model (SSM) with sparse n-gram context mixing and arithmetic coding. The model is initialised from scratch and trained token-by-token on the file being compressed, requiring no pre-trained weights, no GPU, and no external dependencies. The SSM (DM=32, NL=2, approximately 120K active parameters per file) provides a continuously-updated probability estimate over BPE tokens, while nine sparse n-gram hash tables (bigram through 32-gram, 16M slots each) add exact local and long-range pattern memorisation via a softmax-invariant logit-bias mechanism that updates only non-zero-count tokens. An entropy-adaptive scaling mechanism modulates the n-gram contribution based on the SSM's predictive confidence, preventing over-correction when the neural model is already well-calibrated. On the standard enwik8 benchmark, StateSMix achieves 2.123 bpb on 1 MB, 2.149 bpb on 3 MB, and 2.162 bpb on 10 MB, beating xz -9e (LZMA2) by 8.7%, 5.4%, and 0.7% respectively. Ablation experiments establish the SSM as the dominant compression engine: it alone accounts for a 46.6% size reduction over a frequency-count baseline and beats xz without any n-gram component, while n-gram tables provide a complementary 4.1% gain through exact context memorisation. OpenMP parallelisation of the training loop yields 1.9x speedup on 4 cores. The system is implemented in pure C with AVX2 SIMD and processes approximately 2,000 tokens per second on commodity x86-64 hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。