长上下文模型下，精选示例比堆更多示例更有效。
Refract ICL: Rethinking Example Selection in the Era of Million-Token Models
- 通过重复难点示例并用零样本预测作误差信号，优化注意力分配。
- 在数千示例条件下，性能提升显著，尤其适用于输出类别少的任务。
- 为超长上下文模型提供高效示例选择新方案，适合高精度任务场景。
长上下文大语言模型的出现使得数百甚至上千个示范示例用于上下文学习（ICL）成为可能，这在过去是不切实际的。本文研究了传统ICL示例选择策略——在测试输入相似性与示例多样性之间权衡——在使用大量示范时是否依然有效。实验表明，尽管更长的上下文可容纳更多示例，但单纯增加示例数量并不能保证性能提升。智能的ICL选择依然至关重要。为此，我们提出Refract ICL，一种专为长上下文场景设计的新算法，通过战略性重复难点示例并引入零样本预测作为误差信号，引导模型关注关键信息。结果表明，Refract ICL显著提升了如Gemini 1.5 Pro等超长上下文模型的表现，尤其在输出类别较少的任务上效果突出。
原文摘要 · Abstract (English)
The emergence of long-context large language models (LLMs) has enabled the use of hundreds, or even thousands, of demonstrations for in-context learning (ICL) - a previously impractical regime. This paper investigates whether traditional ICL selection strategies, which balance the similarity of ICL examples to the test input (using a text retriever) with diversity within the ICL set, remain effective when utilizing a large number of demonstrations. Our experiments demonstrate that, while longer contexts can accommodate more examples, simply increasing the number of demonstrations does not guarantee improved performance. Smart ICL selection remains crucial, even with thousands of demonstrations. To further enhance ICL in this setting, we introduce Refract ICL, a novel ICL selection algorithm specifically designed to focus LLM attention on challenging examples by strategically repeating them within the context and incorporating zero-shot predictions as error signals. Our results show that Refract ICL significantly improves the performance of extremely long-context models such as Gemini 1.5 Pro, particularly on tasks with a smaller number of output classes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。