用相似任务的示范迁移，提升大模型上下文学习效果。
In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks
- 从相似任务中提取示范，再迁移到目标任务
- 在Super-NI上比从零合成提升2.0%准确率
- 适合需要高质量示范但标注成本高的场景
上下文学习（ICL）通过提供目标任务的示范，帮助大语言模型适应不同任务。由于标注示范成本高，现有方法尝试用大模型自动生成示范，但其质量受限于模型自身能力。受迁移学习启发，本文提出上下文迁移学习（ICTL），通过迁移相似源任务的已标注示范来生成目标任务示范。ICTL包含两步：首先定义优化目标，最小化迁移误差，以筛选与目标任务相近的源示范；然后利用大模型将所选源示范转换为目标任务的定义和格式。在Super-NI数据集上的实验表明，ICTL相比从零生成示范平均提升2.0%，验证了方法的有效性。
原文摘要 · Abstract (English)
In-context learning (ICL) is an effective approach to help large language models (LLMs) adapt to various tasks by providing demonstrations of the target task. Considering the high cost of labeling demonstrations, many methods propose synthesizing demonstrations from scratch using LLMs. However, the quality of the demonstrations synthesized from scratch is limited by the capabilities and knowledge of LLMs. To address this, inspired by transfer learning, we propose In-Context Transfer Learning (ICTL), which synthesizes target task demonstrations by transferring labeled demonstrations from similar source tasks. ICTL consists of two steps: source sampling and target transfer. First, we define an optimization objective, which minimizes transfer error to sample source demonstrations similar to the target task. Then, we employ LLMs to transfer the sampled source demonstrations to the target task, matching the definition and format of the target task. Experiments on Super-NI show that ICTL outperforms synthesis from scratch by 2.0% on average, demonstrating the effectiveness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。