让大模型在推理时动态优化记忆模块，提升长文本理解能力
PERK: Long-Context Reasoning as Test-Time Learning
- 通过测试时梯度更新构建轻量级记忆模块，实现高效长上下文编码
- 在多个任务上比标准微调提升最高20%，跨模型规模表现稳定
- 适合需要强长文本推理的场景，如法律、科研文档分析
长上下文推理需从大量噪声信息中精准识别相关内容。本文提出PERK（基于知识的参数高效推理），一种在测试时通过梯度更新实现长上下文学习的可扩展方法。PERK在元训练阶段采用双层优化：内层快速将上下文编码为低秩适配器（LoRA），作为基础模型的参数高效记忆模块；外层则学习利用更新后的适配器准确回忆和推理长上下文中的相关信息。在多个长上下文推理任务上的评估表明，PERK显著优于标准长上下文微调，在Qwen-2.5（0.5B与7B）上平均绝对性能提升达20%。其优势在不同模型规模和系列间保持一致，甚至可媲美或超越专用长上下文大模型。分析显示，PERK对推理复杂度、长度外推及关键信息位置更具鲁棒性。
原文摘要 · Abstract (English)
Long-context reasoning requires accurately identifying relevant information in extensive, noisy input contexts. In this work, we propose PERK (Parameter Efficient Reasoning over Knowledge), a scalable approach for learning to encode long contexts using gradient updates at test time. Specifically, PERK employs two nested optimization loops in a meta-training phase. The inner loop rapidly encodes contexts into a low-rank adapter (LoRA) that serves as a parameter-efficient memory module for the base model. Concurrently, the outer loop learns to use the updated adapter to accurately recall and reason over relevant information from the encoded long context. Our evaluations on several long-context reasoning tasks show that PERK significantly outperforms the standard long-context finetuning, achieving average absolute performance gains of up to 20% for Qwen-2.5 (0.5B & 7B) on synthetic and real-world long-context reasoning. PERK also maintains its advantages across model scales and families. Compared to specialized long-context LLMs, PERK matches or surpasses their performance. Finally, our analyses show PERK is more robust to reasoning complexity, length extrapolation, and the positions of relevant information in contexts. https://perk-long-context.web.app
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。