用两级大模型压缩长文本，40倍压缩仍保持93%准确率。
Context Cascade Compression: Exploring the Upper Limits of Text Compression
- 小模型先压缩长文本为32-64个隐向量，大模型解码。
- 20倍压缩时准确率达98%，40倍仍保持93%。
- 纯文本流程更简单，或可逼近压缩上限。
长文本输入给大语言模型带来显著的计算与内存挑战。近期DeepSeek-OCR探索了上下文光学压缩的可行性并取得初步成果。受此启发，我们提出上下文级联压缩（C3），以探索文本压缩的上限。方法上，利用两个不同规模的LLM级联处理压缩与解码任务：小模型作为第一阶段，将长上下文压缩为一组隐向量（如32或64个长度），实现高文本令牌到隐向量比率；大模型作为第二阶段，在压缩后上下文上执行解码。实验表明，在20倍压缩比（文本令牌数为隐向量数的20倍）下，模型达到98%解码准确率，远超DeepSeek-OCR约60%的表现；当压缩比提升至40倍时，准确率仍维持在约93%。这表明在上下文压缩领域，C3优于光学字符压缩。该方法采用更简单的纯文本流程，忽略版面、颜色及视觉编码器信息损失等复杂因素，暗示未来光学字符压缩及相关领域可能存在的压缩比上限。代码与模型权重已公开于https://github.com/liufanfanlff/C3-Context-Cascade-Compression。
原文摘要 · Abstract (English)
Million-level token inputs in long-context tasks pose significant computational and memory challenges for Large Language Models (LLMs). Recently, DeepSeek-OCR conducted research into the feasibility of Contexts Optical Compression and achieved preliminary results. Inspired by this, we introduce Context Cascade Compression C3 to explore the upper limits of text compression. Our method cascades two LLMs of different sizes to handle the compression and decoding tasks. Specifically, a small LLM, acting as the first stage, performs text compression by condensing a long context into a set of latent tokens (e.g., 32 or 64 in length), achieving a high ratio of text tokens to latent tokens. A large LLM, as the second stage, then executes the decoding task on this compressed context. Experiments show that at a 20x compression ratio (where the number of text tokens is 20 times the number of latent tokens), our model achieves 98% decoding accuracy, compared to approximately 60% for DeepSeek-OCR. When we further increase the compression ratio to 40x, the accuracy is maintained at around 93%. This indicates that in the domain of context compression, C3 Compression demonstrates superior performance and feasibility over optical character compression. C3 uses a simpler, pure-text pipeline that ignores factors like layout, color, and information loss from a visual encoder. This also suggests a potential upper bound for compression ratios in future work on optical character compression, OCR, and related fields. Codes and model weights are publicly accessible at https://github.com/liufanfanlff/C3-Context-Cascade-Compression
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。