通过对比采样让大模型同时学会上下文和权重内学习，避免能力崩溃。
Training In-Context and In-Weights Mixtures Via Contrastive Context Sampling
- 用相似与随机样本混合构造上下文，促进正确形式的上下文学习。
- 在4个模型上验证，对比上下文使ICL与IWL稳定共存，不退化为复制标签。
- 适合关注大模型多模式学习能力、提升泛化性的研究者使用。
我们研究了协同训练上下文学习（ICL）与权重内学习（IWL）的策略，并实现根据上下文相关性在两者间切换的能力。尽管当前大模型具备两种模式，但标准任务微调常导致ICL退化。本文发现，目标输入与上下文示例之间的相似性结构至关重要：随机上下文会导致ICL与IWL主导失衡，而仅使用相似示例则使ICL退化为不考虑相关性的标签复制。为此，我们提出对比上下文（Contrastive-Context），强制两类对比：（1）在单个上下文中混合相似与随机示例，以演化出正确的ICL形式；（2）在不同上下文中引入不同程度的相似性，以演化ICL-IWL混合。理论分析基于最小模型揭示其必要性。在四个LLM和多个任务上进行充分实证验证。诊断探针表明，对比上下文能稳定维持ICL-IWL混合，避免退化为纯ICL、纯IWL或复制。
原文摘要 · Abstract (English)
We investigate training strategies that co-develop in-context learning (ICL) and in-weights learning (IWL), and the ability to switch between them based on context relevance. Although current LLMs exhibit both modes, standard task-specific fine-tuning often erodes ICL, motivating IC-Train - fine-tuning with in-context examples. Prior work has shown that emergence of ICL after IC-Train depends on factors such as task diversity and training duration. In this paper we show that the similarity structure between target inputs and context examples also plays an important role. Random context leads to loss of ICL and IWL dominance, while only similar examples in context causes ICL to degenerate to copying labels without regard to relevance. To address this, we propose a simple Contrastive-Context which enforces two types of contrasts: (1) mix of similar and random examples within a context to evolve a correct form of ICL, and (2) varying grades of similarity across contexts to evolve ICL-IWL mixtures. We present insights on the importance of such contrast with theoretical analysis of a minimal model. We validate with extensive empirical evaluation on four LLMs and several tasks. Diagnostic probes confirm that contrasted contexts yield stable ICL-IWL mixtures, avoiding collapse into pure ICL, IWL, or copying.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。