用10比特实现从小型到大型模型的高效知识压缩。
Haiku to Opus in Just 10 bits: LLMs Unlock Large Compression Gains
- 通过交互式提问逐步提取关键信息，每问一次仅传1比特。
- 在8个基准上实现0.0006~0.004的压缩比，比之前快100倍。
- 适合需要极低带宽传输大模型能力的场景，如边缘设备部署。
本文研究了大语言模型生成文本在无损与有损压缩下的表现，揭示了一个压缩-计算权衡前沿：更多压缩需更高算力。无损压缩中，领域自适应的LoRA适配器使基于大模型的算术编码压缩效率提升2倍。有损压缩中，通过提示模型生成简洁重写后再编码，可实现约0.03的压缩比，较原始响应压缩提升2倍。我们提出问题提问压缩（QA），一种受‘二十个问题’启发的交互式有损协议：小模型通过向强模型提问是/否问题，每次获得1比特信息。在涵盖数学、科学和代码的8个基准上，10次提问可恢复小模型与大模型间标准任务72%的能力差距，困难任务达38%，压缩比达0.0006至0.004，比现有方法（Deletang et al., 2024）降低超过100倍，表明交互协议远优于直接传输完整响应。
原文摘要 · Abstract (English)
We study the compression of LLM-generated text across lossless and lossy regimes, characterizing a compression-compute frontier where more compression is possible at the cost of more compute. For lossless compression, domain-adapted LoRA adapters can improve LLM-based arithmetic coding by 2x over compression with the base LLM alone. For lossy compression, prompting a model for a succinct rewrite then applying arithmetic coding can achieve compression ratios of approximately 0.03, a 2x improvement over compressing the original response. We further introduce Question-Asking compression (QA), an interactive lossy protocol inspired by the game 'Twenty Questions'. A small model iteratively refines its response by asking yes/no questions to a stronger model, transferring exactly one bit per answer. On 8 benchmarks spanning math, science, and code, 10 binary questions recover 23% to 72% of the capability gap between a small and large model on standard benchmarks and 7% to 38% on harder benchmarks, achieving compression ratios of 0.0006 to 0.004. This is over 100x smaller than prior LLM-based compression (Deletang et al., 2024), suggesting that interactive protocols can transfer knowledge far more efficiently than transmitting full responses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。