基于幂律解码器表示的LLM，用新注意力机制提升推理能力。
PLDR-LLM: Large Language Model from Power Law Decoder Representations
- 用幂律图注意力实现非线性与线性变换，生成推理输出。
- 在80亿词上预训练，零样本和少样本表现媲美同类模型。
- 首次将有向无环图损失用于评估模型特性，可优化性能。
我们提出基于幂律解码器表示的大语言模型（PLDR-LLM），通过幂律图注意力机制实现非线性和线性变换，生成明确的演绎与归纳输出。在约80亿词的RefinedWeb数据集上，使用32的小批量大小对不同层数的PLDR-LLM进行预训练，并表明其在零样本和少样本设置下的表现，与文献中同规模的点积注意力模型相当。研究发现，通过引入有向无环图（DAG）损失作为度量与正则化项，可利用模型的演绎输出来比较模型特性或提升性能。结果表明，初始最大学习率和预热步数对整个预训练过程中演绎输出具有持久影响。本文详细描述了PLDR-LLM的架构、实现及预训练流程。
原文摘要 · Abstract (English)
We present the Large Language Model from Power Law Decoder Representations (PLDR-LLM), a language model that leverages non-linear and linear transformations through Power Law Graph Attention mechanism to generate well-defined deductive and inductive outputs. We pretrain the PLDR-LLMs of varying layer sizes with a small batch size of 32 and $\sim$8B tokens from the RefinedWeb dataset, and show that they achieve competitive performance in zero-shot and few-shot settings compared to scaled dot-product LLMs of similar model size reported in the literature. We show that deductive outputs of PLDR-LLMs can be used to compare model characteristics or improve the performance by introducing the Directed Acyclic Graph (DAG) loss as a metric and regularizer. Our results indicate that the initial maximum learning rate and warm-up steps have a lasting impact on deductive outputs throughout the pretraining. We provide a detailed description of PLDR-LLM architecture, its implementation and the pretraining procedure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。