arXiv:2509.22158cs.CL2025-09被引 2

用可组合的适配器参数化上下文,提升大模型推理效率与稳定性。

Context Parametrization with Compositional Adapters

  • 将上下文信息转化为可代数合并的适配器参数,实现高效推理。
  • 在长上下文和多示例场景下性能优于传统方法,支持超长输入处理。
  • 适配器可逆编码,便于信息恢复与安全控制,适合实际部署。

大型语言模型常通过上下文学习(ICL)或监督微调(SFT)适应新任务,但ICL在多示例时效率低,而SFT带来训练开销且灵活性差。直接将上下文中的指令或示例映射为适配器参数是一种有吸引力的替代方案。现有方法仅基于单一上下文生成适配器,忽略了多段信息的整合需求。为此,我们提出CompAs——一种元学习框架,将上下文翻译为具有组合结构的适配器参数,支持代数合并。该方法带来三重优势:更低的推理成本、长上下文下的更高稳定性,以及在输入超出模型上下文窗口时的合理处理能力。此外,CompAs具备可逆编码能力,能恢复原始输入上下文,提升安全性。在多个选择题和抽取式问答任务上的实验表明,相较于ICL和已有生成式方法,CompAs在输入数量增加时表现更优。本工作确立了可组合适配器生成在扩展大模型应用中的实用价值。

原文摘要 · Abstract (English)

Large language models (LLMs) often seamlessly adapt to new tasks through in-context learning (ICL) or supervised fine-tuning (SFT). However, ICL is inefficient when handling many demonstrations, and SFT incurs training overhead while sacrificing flexibility. Mapping instructions or demonstrations from context directly into adapter parameters offers an appealing alternative. While prior work explored generating adapters based on a single input context, it has overlooked the need to integrate multiple chunks of information. To address this gap, we introduce CompAs, a meta-learning framework that translates context into adapter parameters with a compositional structure that allows them to be merged algebraically. This approach yields three benefits: lower inference cost, improved stability under long contexts, and establishes a principled solution when input exceeds the model's context window. Furthermore, CompAs reversibly encodes information into adapter parameters, enabling recovery of the original input context and facilitating safety. Empirical results on diverse multiple-choice and extractive question answering tasks show that CompAs outperforms ICL and prior generator-based methods, especially when scaling to more inputs. Our work establishes composable adapter generation as a practical and efficient alternative for scaling LLM deployment.

大模型适配器上下文学习可组合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。