用逻辑增长函数扩展上下文长度，提升长文本处理能力。
SELF: Self-Extend the Context Length With Logistic Growth Function
- 通过逻辑函数分组令牌，动态调整不同距离的上下文覆盖范围。
- 在LEval和LongBench上分别提升最高12%和6.4%性能。
- 适合需要长文本理解的场景，如摘要与阅读理解任务。
大型语言模型在处理超出训练上下文长度的长文本时会遇到问题，因为注意力层中的标准位置编码导致相距遥远的标记几乎无法相互影响，从而产生意外结果。为解决此问题，我们提出SELF（基于逻辑增长函数自扩展上下文长度）：一种使用逻辑容量方程对连续标记进行变尺寸分组，并在较短相对距离下保持恒定分组大小的方法。相较于LongLM扩展方法，该模型在LEval上的表现提升高达12%（以Qwen模型为例）；在LongBench的摘要任务中，相比LongLM提升达6.4%（以Llama-2-7b模型为例）；在LEval的阅读理解任务中，性能最高提升5.4%。代码已开源：https://github.com/alexeipc/SELF-LLM。
原文摘要 · Abstract (English)
Large language models suffer issues when operated on long contexts that are larger than their training context length due to the standard position encoding for tokens in the attention layer. Tokens a long distance apart will rarely have an effect on each other and long prompts yield unexpected results. To solve this problem, we propose SELF (Self-Extend the Context Length With Logistic Growth Function): a solution of grouping consecutive tokens at varying group sizes using a logistic capacity equation combined with a constant group size at smaller relative distances. Our model had an increase in performance of up to 12% compared to the LongLM extension method in LEval (specifically on the Qwen model). On summarization related tasks in LongBench, our model performed up to 6.4% better than LongLM (specifically on the Llama-2-7b model). On reading comprehension tasks from LEval, our model performed up to 5.4% better than the LongLM. Our code is available at https://github.com/alexeipc/SELF-LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。