通过合并嵌入空间的连续token，实现大模型长文本高效处理。
Compressing Sequences in the Latent Embedding Space: $K$-Token Merging for Large Language Models
- 在嵌入空间中将连续K个token合并为一个，降低计算开销。
- 最长压缩75%输入长度，性能损失极小，优于现有方法。
- 适合需要处理长文本的场景，如代码编辑与情感分析。
大型语言模型（LLM）在处理长提示时面临显著的计算与内存开销，因自注意力机制随输入长度呈平方级增长。现有提示压缩方法多在词元空间操作，忽视了潜在嵌入空间的效率问题。本文提出K-Token Merging，一种在嵌入空间中压缩的框架：通过轻量编码器将每组连续的K个词元嵌入合并为单一嵌入。压缩序列由经LoRA适配的LLM处理，生成仍保持原始词汇表。在结构推理（Textualized Tree）、情感分类（Amazon Reviews）和代码编辑（CommitPackFT）任务上的实验表明，K-Token Merging位于性能与压缩率的帕累托前沿，可实现最高75%的输入长度压缩，且性能下降极小。代码已开源。
原文摘要 · Abstract (English)
Large Language Models (LLMs) incur significant computational and memory costs when processing long prompts, as full self-attention scales quadratically with input length. Token compression aims to address this challenge by reducing the number of tokens representing inputs. However, existing prompt-compression approaches primarily operate in token space and overlook inefficiencies in the latent embedding space. In this paper, we propose K-Token Merging, a latent-space compression framework that merges each contiguous block of K token embeddings into a single embedding via a lightweight encoder. The compressed sequence is processed by a LoRA-adapted LLM, while generation remains in the original vocabulary. Experiments on structural reasoning (Textualized Tree), sentiment classification (Amazon Reviews), and code editing (CommitPackFT) show that K-Token Merging lies on the Pareto frontier of performance vs. compression, achieving up to 75% input length reduction with minimal performance degradation. Code is available at https://github.com/shsjxzh/K-Token-Merging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。