DeCAL实现文本令牌级压缩，2倍压缩下性能接近原始模型。
DeCAL Tokenwise Compression
- 用去噪预训练的编码器-解码器模型学习高质量压缩表示。
- 2倍压缩时在问答、摘要等任务上表现接近未压缩模型，8倍压缩仅轻微下降。
- 适合需预计算稠密表示的场景，对压缩质量优先的设计友好。
本文提出DeCAL，一种新的令牌级压缩方法。DeCAL采用经过去噪预训练的编码器-解码器语言模型，学习从编码器生成高质量、通用的压缩表示。该方法对编码器进行小幅修改，重点在于最大化压缩质量，即使牺牲计算成本也在所不惜。实验表明，DeCAL在2倍压缩下可匹配未压缩模型在多个下游任务上的表现，且在问答、摘要和多向量检索任务中，压缩至8倍时通常仅出现轻微性能下降。该方法在可复用预计算稠密表示的场景中带来显著节省，我们相信该思路可进一步拓展以适用于更广泛的应用。
原文摘要 · Abstract (English)
This paper introduces DeCAL, a new method for tokenwise compression. DeCAL uses an encoder-decoder language model pretrained with denoising to learn to produce high-quality, general-purpose compressed representations from the encoder. DeCAL applies small modifications to the encoder, with the emphasis on maximizing compression quality, even at the expense of compute. We show that DeCAL at 2x compression can match uncompressed on several downstream tasks, with usually only a minor dropoff in metrics up to 8x compression, among question-answering, summarization, and multi-vector retrieval tasks. DeCAL offers significant savings where pre-computed dense representations can be utilized, and we believe the approach can be further developed to be more broadly applicable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。