给线性循环模型加检索能力，提升上下文记忆效果
Resona: Improving Context Copying in Linear Recurrence Models with Retrieval
- 用外部检索增强线性循环模型的上下文信息获取
- 在多种任务上显著提升语言建模和上下文学习性能
- 适合想改进高效模型上下文理解能力的研究者
大语言模型研究正转向新型架构以挑战主流Transformer模型。线性递归模型因计算效率高成为有力竞争者，但在上下文学习等需从上下文中回忆信息的任务上仍与Transformer有明显差距。本文提出Resona，一种简单且可扩展的框架，为线性递归模型引入检索机制。该方法使模型能从输入上下文中获取并整合相关信息，适应不同任务需求。在多种线性递归模型上的实验表明，Resona增强后的模型在合成任务和真实自然语言任务中均取得显著性能提升，验证了其作为通用方法在改善线性递归大模型上下文学习与语言建模能力方面的有效性。
原文摘要 · Abstract (English)
Recent shifts in the space of large language model (LLM) research have shown an increasing focus on novel architectures to compete with prototypical Transformer-based models that have long dominated this space. Linear recurrent models have proven to be a viable competitor due to their computational efficiency. However, such models still demonstrate a sizable gap compared to Transformers in terms of in-context learning among other tasks that require recalling information from a context. In this work, we introduce Resona, a simple and scalable framework for augmenting linear recurrent models with retrieval. Resona augments models with the ability to integrate retrieved information from the provided input context, enabling tailored behavior to diverse task requirements. Experiments on a variety of linear recurrent models demonstrate that Resona-augmented models observe significant performance gains on a variety of synthetic as well as real-world natural language tasks, highlighting its ability to act as a general purpose method to improve the in-context learning and language modeling abilities of linear recurrent LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。