让小模型也能读懂长文本，通过实时微调提升理解能力。
LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning
- 在测试时动态调整模型参数，适应长输入
- 使Llama 3等短上下文模型可处理任意长度文本
- 无需离线训练，适合快速部署和通用场景
长上下文理解对大语言模型仍是挑战，因其上下文窗口有限。本文提出长输入微调（LIFT）框架，通过在测试时适配模型参数以增强其长上下文任务表现。LIFT无需离线长上下文训练，实现高效长文本处理，可提升任意短上下文模型的长文本理解能力。结合上下文学习与预训练微调，该方法使Llama 3等模型在LooGLE和LongBench等主流长文本基准上性能显著提升。我们还全面分析了LIFT在长上下文理解中的优势与局限，为未来研究提供方向。
原文摘要 · Abstract (English)
Long context understanding remains challenging for large language models due to their limited context windows. This paper introduces Long Input Fine-Tuning (LIFT) for long context modeling, a novel framework that enhances LLM performance on long-context tasks by adapting model parameters to the context at test time. LIFT enables efficient processing of lengthy inputs without the computational burden of offline long-context adaptation, and can improve the long-context capabilities of arbitrary short-context models. The framework is further enhanced by integrating in-context learning and pre-LIFT supervised fine-tuning. The combination of in-context learning and LIFT enables short-context models like Llama 3 to handle arbitrarily long contexts and consistently improves their performance on popular long-context benchmarks like LooGLE and LongBench. We also provide a comprehensive analysis of the strengths and limitations of LIFT on long context understanding, offering valuable directions for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。