用谱方法压缩长提示,提速推理且保持质量。
Spectral-LSH: Sub-Quadratic Prompt Compression via Krylov-Projected Locality-Sensitive Hashing
- 通过克里洛夫投影与随机特征近似注意力核,避免显式计算复杂度。
- 在16倍压缩下,Qwen2.5-14B的PPL从9.533降至3.427。
- 自适应选择分块或谱聚类,兼顾速度与效果,适合长文本任务。
长提示推理成本高昂,因预填充注意力随序列长度呈二次增长。本文提出Spectral-LSH,一种无需训练的提示压缩方法,在提示进入语言模型前执行。该方法利用克里洛夫子空间方法结合随机特征,近似隐式注意力核算子的主要成分,避免了显式的 $O(N^2)$ 注意力核构造。随后在所得注意力特征空间中应用SimHash,对相似标记进行分组并聚合为带因果位置分配的宏标记。我们在C4上评估Mistral-7B-Instruct-v0.3、Qwen2.5-7B-Instruct和Qwen2.5-14B-Instruct。实验显示压缩比存在相变现象:当压缩比 $ρ < 4\times$ 时,局部冗余低,轻量分块通常提供最佳延迟-质量权衡;当 $ρ > 8\times$ 时,谱路径能保留分块丢失的质量。在 $ρ=16\times$ 时,Qwen2.5-7B(自适应)将PPL从353.409降至196.963,而Qwen2.5-14B(自适应)从9.533降至3.427。在包含JSON、代码和表格结构的长上下文压力测试中,$8\times$ 压缩下局部LSH在所有指标上均优于分块。自适应后端通过低压缩时使用分块路径、高压缩时切换至谱聚类,兼顾两种场景,尽管分块在总延迟上仍最快。
原文摘要 · Abstract (English)
Long-prompt inference remains expensive because prefill attention scales quadratically with sequence length. We propose Spectral-LSH, a training-free prompt compression method that operates before the prompt enters the language model. Spectral-LSH approximates the dominant components of an implicit attention-kernel operator using a Krylov subspace method together with random features, avoiding explicit $O(N^2)$ attention-kernel materialization. It then applies SimHash in the resulting attention eigenspace to group similar tokens and aggregate them into macro-tokens with causal positional assignments. We evaluate Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct, and Qwen2.5-14B-Instruct on C4. Our experiments reveal a compression-ratio phase transition. Below $ρ= 4 \times$, local token redundancy is low enough that lightweight chunking typically provides the best latency--quality trade-off. Above $ρ= 8 \times$, the spectral path preserves quality that chunking loses. At $ρ= 16 \times$, Qwen2.5-7B (adaptive) reduces the PPL ratio from 353.409 to 196.963, while Qwen2.5-14B (adaptive) reduces it from 9.533 to 3.427. On a small long-context structured stress test containing JSON-like, code-like, and table-like inputs, local LSH also improves every metric over chunking at $8 \times$. The adaptive backend captures both regimes by using the chunk path at low compression and spectral clustering at high compression, although chunking remains the fastest backend in total latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。