arXiv:2412.16926cs.CLcs.AI2024-12ACL被引 12

长上下文模型下,选例方法不再关键,填满上下文才是提升关键。

Revisiting In-Context Learning with Long Context Language Models

  • 用长上下文模型扩大示例输入空间,突破传统数量限制。
  • 复杂选例方法效果不如随机采样,性能提升仅5%靠数据增强。
  • 适合研究大上下文场景下ICL机制的学者与工程师。

上下文学习(ICL)是语言模型根据输入上下文中的示例进行预测的技术。以往受限于上下文窗口大小,示例选择技术对性能至关重要。然而,长上下文语言模型(LCLMs)的出现大幅增加了可纳入上下文的示例数量,引发疑问:在多示例场景中,ICL性能是否仍受选例方法影响?我们通过在18个涵盖4类任务的数据集上开展广泛实验,发现复杂示例选择方法并未显著优于简单的随机采样。相反,我们发现LCLMs的出现已将ICL的核心挑战从选择有效示例转变为收集足够多的示例以填满上下文窗口。某些数据集即使包含所有可用示例仍未充分利用上下文容量;通过简单数据增强扩充上下文示例,可使ICL性能提升5%。

原文摘要 · Abstract (English)

In-Context Learning (ICL) is a technique by which language models make predictions based on examples provided in their input context. Previously, their context window size imposed a limit on the number of examples that can be shown, making example selection techniques crucial for identifying the maximally effective set of examples. However, the recent advent of Long Context Language Models (LCLMs) has significantly increased the number of examples that can be included in context, raising an important question of whether ICL performance in a many-shot regime is still sensitive to the method of sample selection. To answer this, we revisit these approaches in the context of LCLMs through extensive experiments on 18 datasets spanning 4 tasks. Surprisingly, we observe that sophisticated example selection techniques do not yield significant improvements over a simple random sample selection method. Instead, we discover that the advent of LCLMs has fundamentally shifted the challenge of ICL from that of selecting the most effective examples to that of collecting sufficient examples to fill the context window. Specifically, in certain datasets, including all available examples does not fully utilize the context window; however, by augmenting the examples in context with a simple data augmentation approach, we substantially improve ICL performance by 5%.

上下文学习长上下文数据增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。