用大模型预测代码符号排名，限定范围提升压缩效率
LLM-based Source Code Compression via Thresholded Symbol Ranking
- 限定大模型预测排名在前T位，减少冗余计算
- 相比之前方法压缩率提升37%，速度更快40%
- 适合需要高效存储代码的工程与研究场景
我们研究损失无损的源代码压缩问题，针对大型软件存档（如 Software Heritage）的存储需求。通用压缩器（如 zstd、bzip2）虽平衡压缩比与速度，但未能充分利用源代码的固有规律。近期方法利用大语言模型（LLM）在香农符号排序框架中预测排名，但允许排名无限增长，导致吞吐量大幅下降，且未解决是否必须显式编码所有排名的问题。本文提出两种新型符号排序变体，将预测限制在前 T 位（T=1 或 63），超出阈值的符号作为异常值，与排名流一起由通用压缩器联合压缩。我们在 30 个 LLM 上进行首次大规模评估，涵盖通用型、代码专用型及量化模型。所提 T-受限方法在压缩比上优于以往基于 LLM 的压缩器（最高相对提升 37%），压缩速度提升 40%。相较于 zstd、bzip2 等通用压缩器，最高实现 82% 相对压缩增益，但速度较低，提供了压缩-速度谱中的新权衡点。我们还发现该方法在源代码上表现优于自然语言，表明代码中的规律能被 LLM 捕捉，而通用匹配型压缩器难以识别。最后讨论开放问题，为理论与实践研究提供方向。
原文摘要 · Abstract (English)
We study the problem of lossless compression of source code, motivated by the storage demands of large-scale software archives, such as Software Heritage (https://www.softwareheritage.org/). General-purpose compressors (e.g., zstd, bzip2) offer a good trade-off between compression ratio and speed, but fail to exploit all special regularities inherent in source code. Recent approaches leverage Large Language Models (LLMs) within Shannon's symbol-ranking framework, relying on a scheme in which the predicted rank can grow arbitrarily. While effective at reducing space, this setting incurs significant throughput degradation, and leaves open the question whether it is necessary to explicitly encode all ranks. In this work, we introduce LLM-based compressors deploying two novel symbol-ranking variants that bound predictions to the top-$T$ ranks ($T=1$ or $63$), with out-of-threshold symbols stored as exceptions and compressed jointly with the rank stream via general-purpose compressors. We conduct the first large-scale evaluation of LLM-based source code compression across 30 LLMs, including general-domain, code-specialized, and quantized models. Our $T$-bounded approach outperforms prior LLM-based compressors both in compression ratio (up to 37% relative improvement) and compression throughput (40% faster). Compared to general-purpose compressors (e.g., zstd, bzip2), we obtain up to 82% relative compression gain but at a lower speed, thus offering a new trade-off point in the compression-speed spectrum. We also show that these gains are stronger on source code than on natural language, suggesting an interesting indication, namely that source code exposes regularities captured by LLMs but missed by general-purpose exact-match-based compressors. We conclude by commenting on open problems that offer theoretical and practical avenues of research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。