用内存模型替代大模型，高效省电且结果可解释。
Memory-based Language Models: An Efficient, Explainable, and Eco-friendly Approach to Large Language Modeling
- 基于最近邻检索构建语言模型，全靠CPU运行
- 预测速度接近实时，训练推理碳排放极低
- 适合关注能效与透明性的研究者和开发者
我们提出一种基于内存的语言建模方法，作为深度神经网络语言模型的高效、环保替代方案。该方法具备对数线性可扩展的下一个词预测性能和强大的记忆能力。通过快速近似k近邻分类实现，内存型语言模型在训练和推理阶段均仅依赖CPU，具有极低的能耗和高吞吐量。其内部机制简单透明。我们将实现的OLIFANT模型与GPT-2和GPT-Neo在下一个词预测准确率、估算碳排放及运行速度上进行对比，并进行了深入分析。
原文摘要 · Abstract (English)
We present memory-based language modeling as an efficient, eco-friendly alternative to deep neural network-based language modeling. It offers log-linearly scalable next-token prediction performance and strong memorization capabilities. Implementing fast approximations of k-nearest neighbor classification, memory-based language modeling leaves a relatively small ecological footprint both in training and in inference mode, as it relies fully on CPUs and attains low token latencies. Its internal workings are simple and fully transparent. We compare our implementation of memory-based language modeling, OLIFANT, with GPT-2 and GPT-Neo on next-token prediction accuracy, estimated emissions and speeds, and offer some deeper analyses of the model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。