提出双空间编码架构，解决模型上下文学习与内在学习的冲突。
Reconciling In-Context and In-Weight Learning via Dual Representation Space Encoding
- 将上下文和样本分别编码到任务与样本两个独立表示空间。
- 在少样本分类和伪算术任务中同时提升上下文学习与内在学习性能。
- 理论与实验结合，验证双空间设计能有效调和两类学习能力。
上下文学习（ICL）是基于多样化序列任务预训练的Transformer模型的重要能力。然而，先前研究发现ICL常与模型固有的参数内学习（IWL）能力产生冲突。通过分析一个简化模型在合成实验中学习的表示空间，我们识别出上下文与样本共享的编码空间可能是冲突的根源。为此，我们修改模型架构，将上下文与样本分别编码至两个独立空间：任务表示空间与样本表示空间。在简明而有原则的框架下建模这两个空间，假设线性表示结构，并将其视为一对对偶空间。理论分析与实证结果均表明，所提出的CoQE架构在单值答案设定下有效。它不仅通过改进表示学习提升了ICL性能，还在合成少样本分类与新设计的伪算术任务中成功调和了ICL与IWL能力。代码见：https://github.com/McGuinnessChen/dual-representation-space-encoding
原文摘要 · Abstract (English)
In-context learning (ICL) is a valuable capability exhibited by Transformers pretrained on diverse sequence tasks. However, previous studies have observed that ICL often conflicts with the model's inherent in-weight learning (IWL) ability. By examining the representation space learned by a toy model in synthetic experiments, we identify the shared encoding space for context and samples in Transformers as a potential source of this conflict. To address this, we modify the model architecture to separately encode the context and samples into two distinct spaces: a task representation space and a sample representation space. We model these two spaces under a simple yet principled framework, assuming a linear representational structure and treating them as a pair of dual spaces. Both theoretical analysis and empirical results demonstrate the effectiveness of our proposed architecture, CoQE, in the single-value answer setting. It not only enhances ICL performance through improved representation learning, but also successfully reconciles ICL and IWL capabilities across synthetic few-shot classification and a newly designed pseudo-arithmetic task. Code: https://github.com/McGuinnessChen/dual-representation-space-encoding
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。