一个可调控推理成本的高效语言模型，单模型实现高低质量平衡。
Controllably Efficient Language Models
- 通过压缩序列并注意力解码，用块大小调节质量与成本权衡。
- 在长文本召回任务上媲美10种主流高效模型，推理成本相当。
- 无需重训练即可在测试时切换性能模式，适合不同应用场景。
Transformer中的注意力机制带来显著的推理开销，催生了稀疏注意力、滑动窗口注意力、卷积和线性注意力等高效序列混合器。尽管这些方法大幅降低推理成本，但常以牺牲上下文召回质量为代价。预先在训练时固定这种质量-成本权衡会导致初始即次优：某些下游任务可能需要更强的上下文记忆能力，而另一些则更关注低延迟与低内存。我们提出一种概念简单的元序列混合器——压缩与注意力变压器（Compress & Attend Transformer, CAT）：通过注意力于已压缩的序列块来解码令牌块。压缩与解码均可采用任意现有序列混合器。从压缩序列解码带来计算与内存节省，块大小决定质量-成本权衡的运行点。重要的是，同时在多个块大小下训练CAT，可在测试时无须重训练即实现该权衡的控制，仅需单一模型。以最基础的选择——密集注意力作为混合器实例化，CAT在真实世界长上下文召回任务上，表现媲美10种流行且多样的高效模型（线性、混合、稀疏），推理成本相当，均来自单个训练模型。此外，CAT在长上下文理解基准上表现优异，生成吞吐量比密集Transformer高出1.4至3.7倍。代码见：https://github.com/rajesh-lab/cat-transformer
原文摘要 · Abstract (English)
The substantial inference costs of attention in transformers motivated the development of efficient sequence mixers: namely sparse and sliding window attention, convolutions and linear attention. Although these approaches result in impressive reductions in inference costs, they often trade-off with quality, specifically in-context recall. Apriori fixing this quality-cost tradeoff at training time means being suboptimal from the get-go: some downstream applications might fundamentally require more memory for in-context recall, while other tasks may require lower latency and memory. We propose a conceptually simple meta-sequence mixer with inference-cost controllability: the Compress & Attend Transformer (CAT). CAT decodes chunks of tokens by attending to compressed chunks of the sequence so far. Both compression and decoding can use any existing sequence mixer. Decoding from the compressed sequence yields compute and memory savings, with chunk size setting the operating point on the quality-cost trade-off. Importantly, training CAT across multiple chunk sizes at once unlocks test-time control of this trade-off without any retraining, all in a single model. Instantiated with the most basic choice, dense attention as the mixer, CAT surprisingly suffices to match 10 popular and diverse efficient models (linear, hybrids, sparse) on real-world long-context recall at comparable inference costs, all from a single trained model. CAT further performs competitively on long-context understanding benchmarks while providing 1.4-3.7x higher generation throughput than a dense transformer. Code is at: https://github.com/rajesh-lab/cat-transformer
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。