用梯度匹配选示范，让大模型少调多学更准。
Selecting Demonstrations for Many-Shot In-Context Learning via Gradient Matching
- 通过匹配微调梯度选示范，模拟全量训练效果。
- 在4到128示例下，性能比随机选提升2%~4%。
- 适合想提升少样本学习效果的研究者和工程师。
上下文学习（ICL）使大语言模型无需微调即可快速适应新任务，但其示范选择仍是关键挑战。尽管多示范ICL表现良好，现有方法仍依赖随机选择。由于传统实例级检索不适用于多示范场景，我们假设ICL与微调的数据需求相似。为此，提出一种基于梯度匹配的新方法：通过对比目标任务全训练集与候选示范的微调梯度，选择能逼近全量训练效果的示范。在Qwen2.5-3B、Llama3-8B等小模型上进行梯度匹配后，该方法在更大模型（如Qwen2.5-72B、Llama3-70B）上从4到128示例均优于随机选择，性能提升达4%；在5个闭源大模型上也提升约2%。本工作显著提升了多示范ICL的可靠性与有效性，推动其广泛应用。
原文摘要 · Abstract (English)
In-Context Learning (ICL) empowers Large Language Models (LLMs) for rapid task adaptation without Fine-Tuning (FT), but its reliance on demonstration selection remains a critical challenge. While many-shot ICL shows promising performance through scaled demonstrations, the selection method for many-shot demonstrations remains limited to random selection in existing work. Since the conventional instance-level retrieval is not suitable for many-shot scenarios, we hypothesize that the data requirements for in-context learning and fine-tuning are analogous. To this end, we introduce a novel gradient matching approach that selects demonstrations by aligning fine-tuning gradients between the entire training set of the target task and the selected examples, so as to approach the learning effect on the entire training set within the selected examples. Through gradient matching on relatively small models, e.g., Qwen2.5-3B or Llama3-8B, our method consistently outperforms random selection on larger LLMs from 4-shot to 128-shot scenarios across 9 diverse datasets. For instance, it surpasses random selection by 4% on Qwen2.5-72B and Llama3-70B, and by around 2% on 5 closed-source LLMs. This work unlocks more reliable and effective many-shot ICL, paving the way for its broader application.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。