通过低维投影注意力提升大模型效率与性能，训练快12.4%且准确率更高。
Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention
- 仅在注意力层使用低维投影模块，兼顾效率与效果。
- 30亿参数模型下，推理速度提升12.4%，困惑度降低5%。
- 适合追求高效部署的大模型研究者与工程师。
提升大语言模型（LLMs）的有效性与效率是关键但具挑战性的目标。本文发现，当低秩预训练的参数被精准定位时，其表现可实现可扩展的有效性，而非仅牺牲性能。具体而言，仅将低维模块应用于注意力层，即可同时提升模型效率与性能。我们提出该结构为低维投影注意力（Low-dimensional Projected Attention, LPA），并提供解释性分析。在1.3亿、3.7亿至30亿参数规模上进行大量实验，验证了LPA的有效性与可扩展性。结果表明，相较于原始Transformer，LPA模型在测试困惑度（ppl）上改善约5%，训练时间节省最高达12.4%，且在下游任务中表现更优。
原文摘要 · Abstract (English)
Improving the effectiveness and efficiency of large language models (LLMs) simultaneously is a critical yet challenging research goal. In this paper, we find that low-rank pre-training, normally considered as efficient methods that will compromise performance, can be scalably effective when reduced parameters are precisely targeted. Specifically, applying the low-dimensional module only to the attention layer -- resolves this issue and enhances both effectiveness and efficiency. We refer to this structure as Low-dimensional Projected Attention (LPA) and provide an explanatory analysis. Through extensive experimentation at parameter scales of 130M, 370M, and scaling up to 3B, we have validated the effectiveness and scalability of LPA. Our results show that LPA model can save up to 12.4% in time while achieving an approximate 5% improvement in test perplexity (ppl) and on downstream tasks compared with the vanilla Transformer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。