LoRA作为解码时参数记忆，在上下文丢失时仍能提升文档问答效果。
Rethinking LoRA Memory Through the Lens of KV Cache Compression
- 将文档信息存入轻量LoRA适配器，作为解码阶段的参数记忆。
- 当上下文完全丢失时，LoRA可恢复13-21个ROUGE-L点，显著优于仅靠缓存。
- 使用问答监督训练的LoRA性能远超原始文本预测，适合高效文档检索场景。
参数化检索增强通过将文档信息编码为轻量级、文档专属的模块（如LoRA适配器），减少将全部证据作为输入上下文的需求。然而，参数记忆与存储于KV缓存中的上下文记忆之间的交互尚不清晰。本文通过逐步清除文档的键值对状态，测量在保留上下文之外，文档LoRA何时仍具价值。结果表明，当KV缓存基本完整时，文档LoRA作用有限；但在高压压缩下，其价值凸显——当无文档上下文留存时，可恢复13-21个ROUGE-L点。该增益在基础模型已编码文档、适配器仅用于生成阶段时最大，提示文档LoRA更应被视为解码时的参数记忆，而非文档编码器。此外，基于问答任务的监督训练产生的适配器显著强于仅依赖原始上下文的自回归预测。这些发现表明，文档LoRA是一种互补的记忆通道，其价值恰在上下文缺失时显现。
原文摘要 · Abstract (English)
Parametric retrieval augmentation encodes document information into lightweight, document-specific modules such as LoRA adapters, reducing the need to include all evidence as input context. However, it remains unclear how this parameter-side memory interacts with context-side memory stored in the KV cache. We study this interaction in document-level question answering by progressively evicting document key-value states and measuring when a document LoRA contributes beyond the retained context. We find that document LoRA adds little when the KV cache is largely intact, but becomes increasingly useful under aggressive compression, recovering 13-21 ROUGE-L points when no document context remains. The gain is largest when the base model encodes the document, and the adapter is applied only during answer generation, suggesting that document LoRA is better understood as decoding-time parametric memory than as a document encoder. Finally, QA-style supervision produces substantially stronger adapters than raw-context next-token-prediction. These results position document LoRA as a complementary memory channel whose value emerges precisely when context-side evidence is scarce.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。