PSC通过相位校准扩展大模型上下文窗口，提升长文本理解能力。
PSC: Extending Context Window of Large Language Models via Phase Shift Calibration
- 引入小模块PSC校准RoPE频率，无需重新搜索最优因子。
- 在16k、32k、64k上下文下，困惑度下降显著提升。
- 兼容多种模型与任务，适用于长文本生成与理解场景。
旋转位置编码（RoPE）是众多大语言模型中广泛采用的高效位置编码方法。近期许多方法尝试基于RoPE扩展上下文窗口，核心思路是预定义或搜索一组因子来重缩放RoPE的基础频率。然而，由于指数级搜索空间，现有方法难以预设最优因子。为此，我们提出PSC（Phase Shift Calibration），一个用于校准现有方法预设频率的小模块。实验表明，结合PSC可显著提升多种方法（如PI、YaRN、LongRoPE）的表现。在多个模型与任务上的广泛实验显示：(1) 启用PSC后，随着上下文窗口从16k增至32k和64k，困惑度的相对下降幅度持续增加；(2) 本方法具备广泛适用性，并在多种模型与任务中表现出强鲁棒性。代码已公开于https://github.com/WNQzhu/PSC。
原文摘要 · Abstract (English)
Rotary Position Embedding (RoPE) is an efficient position encoding approach and is widely utilized in numerous large language models (LLMs). Recently, a lot of methods have been put forward to further expand the context window based on RoPE. The core concept of those methods is to predefine or search for a set of factors to rescale the base frequencies of RoPE. Nevertheless, it is quite a challenge for existing methods to predefine an optimal factor due to the exponential search space. In view of this, we introduce PSC (Phase Shift Calibration), a small module for calibrating the frequencies predefined by existing methods. With the employment of PSC, we demonstrate that many existing methods can be further enhanced, like PI, YaRN, and LongRoPE. We conducted extensive experiments across multiple models and tasks. The results demonstrate that (1) when PSC is enabled, the comparative reductions in perplexity increase as the context window size is varied from 16k, to 32k, and up to 64k. (2) Our approach is broadly applicable and exhibits robustness across a variety of models and tasks. The code can be found at https://github.com/WNQzhu/PSC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。