让小模型理解长文本，通过动态调参突破上下文限制。
LIFT: A Novel Framework for Enhancing Long-Context Understanding of LLMs via Long Input Fine-Tuning
- 用合成任务微调模型参数，让小模型自适应处理长输入。
- 8k上下文下首字延迟低于10秒，效率远超传统长上下文模型。
- 适合资源有限但需长文本理解的部署场景，如边缘计算。
由于上下文窗口受限，大模型在长文本理解方面仍面临挑战。本文提出长输入微调（LIFT）框架，通过将长输入动态编码至模型参数中，使任意短上下文大模型能够理解超出其原始上下文长度的信息。与不断扩展上下文窗口不同，LIFT将输入信息“吸收”进参数，避免了标准长上下文模型中二次增长的输入复杂度。不同于简单的长期预训练，LIFT采用精心设计的大模型生成的合成任务，提升模型理解能力而非仅记忆。为缓解微调开销，我们构建了高度优化的流水线，在8k上下文下将首次生成时间（TTFT）控制在10秒以内。本文还分析了LIFT的优势与局限性，探讨了大规模部署可行性，并指明未来研究方向。代码已开源：https://github.com/MuLabPKU/LIFT。
原文摘要 · Abstract (English)
Long-context understanding remains challenging for LLMs due to limited context windows. This paper introduces Long Input Fine-Tuning (LIFT), a framework that improves the long-context performance of arbitrary short-context LLMs by dynamically adapting their parameters to each long input. Instead of endlessly extending context windows to fit longer inputs in context, LIFT stores and absorbs the input in parameters. By fine-tuning long inputs into parameters, LIFT enables short-context LLMs to answer questions even when required information is absent from the inference context, avoiding the quadratic input-length complexity of standard long-context models. Rather than simple continued pretraining on new long contexts, LIFT uses carefully designed LLM-generated synthetic tasks to enhance comprehension beyond memorization. To offset fine-tuning overhead, we design a highly optimized pipeline that reduces Time to First Token (TTFT) to under 10 seconds for 8k context. We further analyze LIFT's strengths and limitations, discuss large-scale deployment feasibility, and highlight future research directions. Implementation is open-sourced at https://github.com/MuLabPKU/LIFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。