让Transformer用少量标签+大量无标签数据提升学习效果
In-Context Semi-Supervised Learning
- 利用少量有标签+大量无标签上下文进行学习
- 在标签极少时仍能实现高准确率预测
- 适合研究大模型如何利用无标签数据
近年来对Transformer在上下文学习（ICL）中的能力研究日益增多，但现有理论多聚焦于显式标注样本的监督学习。实践中，即使标签稀疏或缺失，Transformer仍表现良好，表明未标记上下文存在重要结构。本文提出并研究了上下文半监督学习（IC-SSL），即少量有标签样本搭配大量无标签点，证明Transformer可利用无标签上下文构建鲁棒、依赖上下文的表示，从而实现准确预测，并显著提升低标签场景下的性能，为理解Transformer如何在ICL框架内利用无标签上下文进行表征学习提供了基础洞见。
原文摘要 · Abstract (English)
There has been significant recent interest in understanding the capacity of Transformers for in-context learning (ICL), yet most theory focuses on supervised settings with explicitly labeled pairs. In practice, Transformers often perform well even when labels are sparse or absent, suggesting crucial structure within unlabeled contextual demonstrations. We introduce and study in-context semi-supervised learning (IC-SSL), where a small set of labeled examples is accompanied by many unlabeled points, and show that Transformers can leverage the unlabeled context to learn a robust, context-dependent representation. This representation enables accurate predictions and markedly improves performance in low-label regimes, offering foundational insights into how Transformers exploit unlabeled context for representation learning within the ICL framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。