让预训练语言模型通过递归结构更深度思考，提升效率与性能。
Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
- 用渐进式递归课程提升模型有效深度，训练时保持性能
- 在数学任务上，同等算力下性能优于原模型微调
- 适合关注推理效率与模型压缩的研究者
近期深度递归语言模型的研究表明，递归可将训练时的计算量和参数量与推理时的计算量解耦。本文研究如何将现有的非递归预训练语言模型转化为深度递归模型。我们发现，通过在训练过程中采用递归层级的课程策略逐步增加模型的有效深度，可在保持性能的同时降低整体计算成本。实验结果表明，在数学任务上，将预训练模型转换为递归模型后，在给定算力预算下表现优于对原始非递归模型进行微调的方式。
原文摘要 · Abstract (English)
Recent advances in depth-recurrent language models show that recurrence can decouple train-time compute and parameter count from test-time compute. In this work, we study how to convert existing pretrained non-recurrent language models into depth-recurrent models. We find that using a curriculum of recurrences to increase the effective depth of the model over the course of training preserves performance while reducing total computational cost. In our experiments, on mathematics, we observe that converting pretrained models to recurrent ones results in better performance at a given compute budget than simply post-training the original non-recurrent language model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。