提出新方法让大模型在不同例子顺序下表现一致，且效果更好。
Rethinking Invariance in In-context Learning
- 设计新算法同时避免信息泄露和依赖上下文顺序
- 在多数数据集上超越现有方法，长输入表现更优
- 适合需要稳定推理的场景，如医疗、法律等高风险领域
自回归大语言模型的上下文学习（ICL）虽具重要能力，但对示例顺序敏感，即使示例彼此独立。现有部分方法虽实现顺序不变性，但性能不及标准ICL。本文发现实现不变性的两个关键：信息不泄露与上下文互依性，二者尚未被任何现有方法同时满足。为此提出新方法InvICL，兼具不变性与高效性。实验证明，InvICL在多数基准数据集上优于以往所有方法，尤其在不同输入长度下展现更强泛化能力。代码已开源。
原文摘要 · Abstract (English)
In-Context Learning (ICL) has emerged as a pivotal capability of auto-regressive large language models, yet it is hindered by a notable sensitivity to the ordering of context examples regardless of their mutual independence. To address this issue, recent studies have introduced several variant algorithms of ICL that achieve permutation invariance. However, many of these do not exhibit comparable performance with the standard auto-regressive ICL algorithm. In this work, we identify two crucial elements in the design of an invariant ICL algorithm: information non-leakage and context interdependence, which are not simultaneously achieved by any of the existing methods. These investigations lead us to the proposed Invariant ICL (InvICL), a methodology designed to achieve invariance in ICL while ensuring the two properties. Empirically, our findings reveal that InvICL surpasses previous models, both invariant and non-invariant, in most benchmark datasets, showcasing superior generalization capabilities across varying input lengths. Code is available at https://github.com/PKU-ML/InvICL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。