用大模型压缩实时文本,平衡传输速度与压缩效率。
Real-Time Text Transmission via LLM-Based Entropy Coding over Fixed-Rate Channels
- 用大模型预测字符序列,再用熵编码压缩成可变长度码流。
- 30亿参数模型比1亿参数模型每字符少38%比特,显著提升压缩率。
- 哈夫曼编码适合带宽充裕场景,算术编码压缩最好但延迟高。
学习、预测与压缩紧密相关:能准确预测序列中下一个符号的模型,可与信源编码器结合,将序列压缩至信息论极限附近。当以固定速率读取的分词字符被编码为可变长度码字并流经固定速率信道时,会形成队列,每个字符的延迟取决于码长的均值与方差,以及编码器的算法延迟。本文研究在因果语言模型作为预测器的‘先预测后编码’架构下,实时文本传输中的压缩-延迟权衡问题。比较了多种编码方案:香农(理想)、霍夫曼、算术编码、不同块大小的rANS和gzip。分析区分了编码器固有的算法延迟与随硬件进步而减小的计算延迟。霍夫曼编码在带宽过载通道中是实用选择,算法延迟为零,压缩开销适中。算术编码虽有解码延迟,但能达到近似最优压缩。结果在两种规模下验证:GPT-2(1.24亿参数)和Llama~3.2(30亿参数),参数量相差25倍。该缩放使每字符比特数降低约38%,有效过载信道,从而改变最优编码器的选择。
原文摘要 · Abstract (English)
Learning, prediction, and compression are intimately connected: a model that accurately predicts the next symbol in a sequence can be coupled with a source coder to compress that sequence near its information-theoretic limit. When tokenized characters arriving at a fixed reading pace are encoded into variable-length codewords and streamed over a fixed-rate channel, a queue forms whose per-token delay depends on the mean and variance of the bit lengths and on the coder's algorithmic latency. This paper investigates the compression--delay tradeoff that arises when a causal language model serves as the sequential predictor within a predict-then-code architecture for real-time text transmission. Several coding schemes are compared: Shannon (ideal), Huffman, arithmetic coding, rANS at various block sizes, and gzip. The analysis separates algorithmic delay, inherent to the coder, from computational delay, which shrinks as hardware improves. Huffman is the practical choice for over-provisioned channels, with zero algorithmic delay and modest compression overhead. Arithmetic coding achieves near-optimal compression at the cost of decodability delay. Findings are validated across two scales: GPT-2 (124M) and Llama~3.2 (3B), a twenty-five-fold parameter range. This scaling yields an approximately 38\% reduction in bits per character, effectively over-provisioning the channel and thereby changing which coder is optimal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。