让解码器型模型支持超长上下文，效果媲美两倍长度模型
On The Adaptation of Unlimiformer for Decoder-Only Transformers
- 改造Unlimiformer使其适配解码器模型，通过向量检索优化注意力计算
- 在摘要任务上表现接近双倍上下文长度模型，支持100k上下文输入
- 适用于需要长文本理解的场景，如指令微调模型和自由问答
当前大型语言模型普遍面临上下文长度受限的问题。尽管GPT-4和Claude 2等闭源模型已支持8k/32k和100k上下文，但主流开源模型如LLama-2仍多为4k或更短。Unlimiformer（Bertsch et al., 2023）是一种近期流行的向量检索增强方法，通过kNN索引将交叉注意力计算卸载；但其原生不兼容解码器型变压器。本文研究了将Unlimiformer适配至解码器模型的实际挑战，并提出一系列改进方案。实验扩展至摘要任务之外，新增自由形式问答任务及自定义6.7B GPT指令微调模型。结果表明，在摘要任务上，改进后的模型性能可与拥有2倍上下文长度的模型相当，达到100k上下文处理能力。同时讨论了自由问答与指令微调模型下的局限性与未来方向。
原文摘要 · Abstract (English)
One of the prominent issues stifling the current generation of large language models is their limited context length. Recent proprietary models such as GPT-4 and Claude 2 have introduced longer context lengths, 8k/32k and 100k, respectively; however, despite the efforts in the community, most common models, such as LLama-2, have a context length of 4k or less. Unlimiformer (Bertsch et al., 2023) is a recently popular vector-retrieval augmentation method that offloads cross-attention computations to a kNN index. However, its main limitation is incompatibility with decoder-only transformers out of the box. In this work, we explore practical considerations of adapting Unlimiformer to decoder-only transformers and introduce a series of modifications to overcome this limitation. Moreover, we expand the original experimental setup on summarization to include a new task (i.e., free-form Q&A) and an instruction-tuned model (i.e., a custom 6.7B GPT model). Our results showcase the effectiveness of these modifications on summarization, performing on par with a model with 2x the context length. Moreover, we discuss limitations and future directions for free-form Q&A and instruction-tuned models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。