去掉MLP层的注意力压缩器能更高效地压缩提示，提升生成质量。
Better Prompt Compression Without Multi-Layer Perceptrons
- 用仅保留注意力机制的结构替代原模型，构建轻量压缩器。
- 在480倍压缩比下仍优于传统LoRA压缩器，生成更准确。
- 证明压缩器无需与解码器同构，为后续设计开辟新路径。
提示压缩是一种在不改变生成模型的前提下加速语言模型推理的有前景方法。先前工作使用低秩适配（LoRA）训练的编码器，将提示压缩为学习得到的短序列。然而，我们发现编码器无需保留原始语言模型的架构即可实现有效压缩。本文提出仅注意力压缩器（AOC），通过移除语言模型Transformer块中的多层感知机（MLP）层，构建一个参数量约为原模型67%的编码器。令人惊讶的是，在高达480倍的压缩比范围内，AOC均能更优地重构提示，性能优于未移除MLP层的基线LoRA压缩编码器。结果表明，提示压缩编码器的架构不必与原始解码器一致，为提示压缩的架构探索提供了新方向。
原文摘要 · Abstract (English)
Prompt compression is a promising approach to speeding up language model inference without altering the generative model. Prior works compress prompts into smaller sequences of learned tokens using an encoder that is trained as a LowRank Adaptation (LoRA) of the inference language model. However, we show that the encoder does not need to keep the original language model's architecture to achieve useful compression. We introduce the Attention-Only Compressor (AOC), which learns a prompt compression encoder after removing the multilayer perceptron (MLP) layers in the Transformer blocks of a language model, resulting in an encoder with roughly 67% less parameters compared to the original model. Intriguingly we find that, across a range of compression ratios up to 480x, AOC can better regenerate prompts and outperform a baseline compression encoder that is a LoRA of the inference language model without removing MLP layers. These results demonstrate that the architecture of prompt compression encoders does not need to be identical to that of the original decoder language model, paving the way for further research into architectures and approaches for prompt compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。