让大模型在推理时扩展长文本处理能力，不增加内存开销。
ETT: Expanding the Long Context Understanding Capability of LLMs at Test-Time
- 通过输入分块重叠微调，实现测试时上下文扩展。
- 将GPT-Large和Phi-2的上下文从1k扩展到32k，准确率提升30%。
- 仅微调第二层前馈网络，效果优于全模型微调。
基于Transformer的语言模型在序列长度增长时，计算与内存开销呈平方级上升，难以处理长序列。本文提出 ewmodelacronym（Extend at Test-Time），一种在测试时扩展短上下文Transformer类大模型上下文长度的方法，具备恒定内存需求和线性计算开销。ETT通过将输入上下文切分为重叠的小子序列，并高效微调模型参数来实现上下文扩展。我们在LongBench上评估，将GPT-Large和Phi-2的上下文长度扩展至32倍（从1k增至32k tokens），准确率最高提升30%。我们还研究了如何在模型权重中有效存储上下文信息。通过详尽消融实验发现，仅微调前馈网络的第二层比全模型微调更有效，进一步提升了模型性能。
原文摘要 · Abstract (English)
Transformer-based Language Models' computation and memory overhead increase quadratically as a function of sequence length. The quadratic cost poses challenges when employing LLMs for processing long sequences. In this work, we introduce \ourmodelacronym~(Extend at Test-Time), method for extending the context length of short context Transformer-based LLMs, with constant memory requirement and linear computation overhead. ETT enable the extension of the context length at test-time by efficient fine-tuning the model's parameters on the input context, chunked into overlapping small subsequences. We evaluate ETT on LongBench by extending the context length of GPT-Large and Phi-2 up to 32 times, increasing from 1k to 32k tokens. This results in up to a 30 percent improvement in the model's accuracy. We also study how context can be stored in LLM's weights effectively and efficiently. Through a detailed ablation study, we examine which Transformer modules are most beneficial to fine-tune at test-time. Interestingly, we find that fine-tuning the second layer of the FFNs is more effective than full fine-tuning, leading to a further improvement in the models' accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。