让模型通过循环迭代提升预测精度，不增参数也不需更多数据。
Loop Neural Networks for Parameter Sharing
- 通过残差连接反复循环部分模型，实现逐轮优化预测
- 在GPT-2基础上改进，语言建模性能提升且参数量不变
- 适合追求高精度但受限于计算资源的场景
大型语言模型如GPT的成功得益于其高效预测序列中下一个词元的能力。然而，这些模型在预测任意词元时均需恒定计算量，缺乏迭代优化能力。本文提出一种新型环状神经网络（Loop Neural Network），通过多次迭代处理输入，在不增加模型规模的前提下提升性能。该方法利用残差连接对模型的部分子结构进行反复计算，逐步优化预测结果。实验对比了GPT-2与本方法的变体，在语言建模任务中表现出更优效果，且参数数量相近。重要的是，性能提升无需额外训练数据。
原文摘要 · Abstract (English)
The success of large-scale language models like GPT can be attributed to their ability to efficiently predict the next token in a sequence. However, these models rely on constant computational effort regardless of the complexity of the token they are predicting, lacking the capacity for iterative refinement. In this paper, we introduce a novel Loop Neural Network, which achieves better performance by utilizing longer computational time without increasing the model size. Our approach revisits the input multiple times, refining the prediction by iteratively looping over a subset of the model with residual connections. We demonstrate the effectiveness of this method through experiments comparing versions of GPT-2 with our loop models, showing improved performance in language modeling tasks while maintaining similar parameter counts. Importantly, these improvements are achieved without the need for extra training data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。