用压缩数据训练语言模型，推理时却用原始字节，提升效率还保持鲁棒性。
Proxy Compression for Language Modeling
- 训练时同时学原始字节和外部压缩数据，让模型内建两者映射关系。
- 在代码建模任务中，相同算力下性能显著优于纯字节基线。
- 适合追求高效训练且需字节级推理的场景，尤其大模型更受益。
现代语言模型几乎全部基于固定分词器生成的标记序列进行训练，该分词器通常是对UTF-8字节序列的无损压缩，从而将模型与特定压缩器绑定。本文提出代理压缩（Proxy Compression），一种替代训练方案：在保持压缩输入效率优势的同时，推理阶段提供端到端的原始字节接口。训练过程中，单一语言模型联合学习原始字节序列和外部压缩器生成的压缩视图；通过这一过程，模型学会内部对齐压缩序列与原始字节。这种对齐使得两种格式间具备强迁移能力，即使训练主要使用压缩输入（推理时丢弃）。大量实验表明，在代码语言建模任务中，代理压缩显著提升训练效率，并在固定计算预算下显著超越纯字节级基线。随着模型规模增大，这些优势愈发明显，代理训练模型最终可达到甚至超过分词器方法的表现，同时仅依赖原始字节输入，并保留字节级建模的固有鲁棒性。代码已开源：https://github.com/LZhengisme/proxy-compression。
原文摘要 · Abstract (English)
Modern language models are trained almost exclusively on token sequences produced by a fixed tokenizer, an external lossless compressor often over UTF-8 byte sequences, thereby coupling the model to that compressor. This work introduces proxy compression, an alternative training scheme that preserves the efficiency benefits of compressed inputs while providing an end-to-end, raw-byte interface at inference time. During training, a single language model is jointly trained on raw byte sequences and compressed views generated by external compressors; through the process, the model learns to internally align compressed sequences and raw bytes. This alignment enables strong transfer between the two formats, even when training predominantly on compressed inputs that are discarded at inference. Extensive experiments on code language modeling demonstrate that proxy compression substantially improves training efficiency and significantly outperforms pure byte-level baselines given fixed compute budgets. As model scale increases, these gains become more pronounced, and proxy-trained models eventually match or surpass tokenizer approaches, all while operating solely on raw bytes and retaining the inherent robustness of byte-level modeling. Our code is available at https://github.com/LZhengisme/proxy-compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。