用大模型压缩文本，实现高效长上下文处理。
Large Language Model as Token Compressor and Decompressor
- 将大模型改造成可学习的文本压缩/解压器，生成紧凑的Z-token序列。
- 在多个数据集上保持高质量重建与下游任务性能，降低内存和延迟。
- 适合需要长文本推理的场景，如问答、摘要和生成任务。
本文研究如何将现成的大语言模型（LLM）改造为离散、可变长度的标记压缩器与解压器，以支持长上下文处理。为此，我们设计了一种自表达的自动编码框架，通过轻量级LoRA适配器微调预训练模型，将长文本映射为紧凑的、学习得到的潜在代码序列（称为Z-tokens），并能将其还原为自然语言或任务输出。该表示具有内容自适应性：信息密度高或可预测性低的部分分配更多Z-tokens，冗余区域则通过预算感知的长度正则化更紧凑地表示。我们在Wikipedia、CNN/DailyMail、HotpotQA和QuALITY等长上下文数据集上评估，结果表明该方法在保持重建质量与下游性能的同时，显著减少了有效上下文长度、生成阶段内存占用和端到端延迟。该设计支持直接从压缩上下文解码，也支持在Z-token空间中自回归生成，为高效长上下文推理提供了实用接口。
原文摘要 · Abstract (English)
In this paper, we study whether an off-the-shelf LLM can be adapted into a discrete, variable-length token compressor and decompressor for long-context processing. To this end, we design a self-expressive autoencoding framework that fine-tunes a pretrained LLM with lightweight LoRA adapters to map long texts into compact sequences of learned latent codes, termed Z-tokens, and to decode them back into natural language or task outputs. The resulting representation is content-adaptive: less predictable or information-dense segments can receive more Z-tokens, while redundant regions can be represented more compactly through a budget-aware length regularizer. Our method is evaluated on long-context datasets such as Wikipedia, CNN/DailyMail, HotpotQA, and QuALITY, showing that it preserves reconstruction quality and downstream performance while reducing effective context length, generation-stage memory usage, and end-to-end latency. This simple design supports both direct decoding from compressed contexts and autoregressive generation in the Z-token space, providing a practical interface for efficient long-context inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。