用对数压缩输入 tokens，让 Transformer 轻松处理更长文本。
Gradual Forgetting: Logarithmic Compression for Extending Transformer Context Windows
- 对输入序列做尺度不变的对数压缩，不改动模型结构。
- 在 WikiText-103 和 PG-19 上降低困惑度，且上下文越长效果越好。
- 适合想提升长文本建模能力又不想改架构的研究者。
现有长上下文处理方法多通过引入循环或辅助记忆模块来增加 Transformer 的复杂度。本文提出一种新思路：不修改模型结构，而是直接对输入表示进行改造。受人类记忆认知模型启发，该方法对输入 token 序列施加尺度不变的对数压缩，生成的压缩表示由标准 Transformer 处理，保持架构简洁。在 WikiText-103 与 PG-19 语言建模基准上评估，相比未压缩基线，该方法显著降低困惑度；且随着压缩后时间上下文长度增加，性能持续提升，表明输入级对数压缩是一种简单而有效的扩展 Transformer 长程记忆能力的方法。
原文摘要 · Abstract (English)
Most approaches to long-context processing increase the complexity of the transformer's internal architecture by integrating mechanisms such as recurrence or auxiliary memory modules. In this work, we introduce an alternative approach that modifies the input representation itself, rather than the transformer architecture. Inspired by cognitive models of human memory, our method applies a scale-invariant logarithmic compression to the input tokens. The resulting compressed representation is processed by a standard, unmodified transformer, preserving architectural simplicity. We evaluate this approach on the WikiText-103 and PG-19 language modeling benchmarks, showing a reduction in perplexity compared to uncompressed baselines. Moreover, performance improves consistently with longer compressed temporal contexts, showing that input-level logarithmic compression is a simple and effective way to extend a transformer's long-range memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。