用蒙特卡洛采样研究上下文示例的呈现影响，发现选例数量和顺序关系复杂。
Monte Carlo Sampling for Analyzing In-Context Examples
- 通过蒙特卡洛采样系统分析示例数量、顺序与选择的交互影响
- 不同示例组合下，单样本设置未必优于零样本，性能依赖具体示例
- 尝试用数据估值法选优示例，结果反而比随机采样更差
先前研究表明，上下文学习对示例的顺序、数量和选取方式敏感。但基于消融实验的示例数量指导可能忽略这些因素间的相互作用。本文提出一种基于蒙特卡洛采样的方法，在显式考虑顺序和示例选择的前提下，分析示例数量的影响。结果发现，以往关于应使用多少示例的建议无法在不同示例集和排序下通用；单样本设置是否优于零样本高度依赖于所选示例。此外，受数据估值启发，我们利用该采样方法筛选在多种排序下表现稳定的示例，却发现其性能反而低于随机采样，出现意外退化。
原文摘要 · Abstract (English)
Prior works have shown that in-context learning is brittle to presentation factors such as the order, number, and choice of selected examples. However, ablation-based guidance on selecting the number of examples may ignore the interplay between different presentation factors. In this work we develop a Monte Carlo sampling-based method to study the impact of number of examples while explicitly accounting for effects from order and selected examples. We find that previous guidance on how many in-context examples to select does not always generalize across different sets of selected examples and orderings, and whether one-shot settings outperform zero-shot settings is highly dependent on the selected example. Additionally, inspired by data valuation, we apply our sampling method to in-context example selection to select examples that perform well across different orderings. We find a negative result, that while performance is robust to ordering and number of examples, there is an unexpected performance degradation compared to random sampling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。