提出双解码器架构，实现高效训练与推理
Block-Based Double Decoders
- 用双重因果块注意力实现全监督训练
- 推理时内存和计算减少至少三分之二
- 适合追求高效推理的大型模型部署
编码器-解码器模型在推理时比仅解码器模型节省大量计算，但其预训练目标存在监督稀疏和序列长度动态变化的问题，难以大规模应用。本文提出基于块的双解码器架构，通过双重因果块注意力掩码实现全损失监督和固定序列打包，结合了仅解码器的训练效率与编码器-解码器的推理效率。在缩放定律实验中，该模型显著优于传统编码器-解码器，并在不同规模下接近仅解码器模型的表现。推理时，其KV缓存内存和每标记计算量至少减少2/3，同时保留填充缓存及其他现有优化技术。
原文摘要 · Abstract (English)
Encoder-decoder models offer substantial inference-time savings over decoder-only models, but their pretraining objectives suffer from sparse supervision and dynamic sequence lengths, keeping them out of practice at scale. We propose block-based double decoders, a novel transformer architecture that utilizes doubly-causal block-based attention masks to train with full loss supervision and static sequence packing, combining decoder-only training efficiency with encoder-decoder inference efficiency. In scaling law experiments, block-based double decoders strongly outperform encoder-decoders and closely track decoder-only models across scales. At inference time, they cut KV-cache memory and per-token compute by at least 2/3 without sacrificing prefill caching or other existing inference optimizations available to decoder-only models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。