解决标注不均衡对上下文学习的负面影响，提升模型性能。
Exploring Imbalanced Annotations for Effective In-Context Learning
- 提出RCB方法，同时考虑类别权重与类内特征偏倚进行演示重加权。
- 在多个基准上使现有选择方法平均准确率提升最高5.42%。
- 特别适合处理真实数据中长尾分布下的上下文学习任务。
大型语言模型通过上下文学习（ICL）在下游任务中表现优异，其性能高度依赖于从标注数据集中选取的示范样本。然而，现实场景中的数据集常呈现长尾类别分布，导致示范选择产生偏差。本文发现，此类类别不平衡会显著降低各类任务上的ICL性能，且无论采用何种选择方法均存在此问题。传统重平衡方法仅关注类别权重，忽视了类内特征分布的偏倚，因而效果不佳。为此，我们提出重加权条件偏倚（RCB）方法，通过一个平衡子集估计条件偏倚，并结合类别权重与条件偏倚对示范得分进行重加权。该方法有效避免了对主导类别的过度选择，同时保留了现有选择方法的效率。大量实验证明，本方法可使当前选择方法的平均准确率提升最高达5.42%。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown impressive performance on downstream tasks through in-context learning (ICL), which heavily relies on the demonstrations selected from annotated datasets. However, these datasets often exhibit long-tailed class distributions in real-world scenarios, leading to biased demonstration selection. In this work, we show that such class imbalances significantly degrade the ICL performance across various tasks, regardless of selection methods. Moreover, classical rebalancing methods, which focus solely on class weights, yield poor performance due to neglecting condition bias--skewed feature distributions within classes. To address this, we propose Reweighting with Conditional Bias (dubbed RCB), a simple and complementary approach to enhance ICL performance under class imbalance. In particular, RCB estimates conditional bias using a balanced subset and re-weights demonstration scores based on both class weight and conditional bias. In effect, RCB prevents over-selection from dominant classes while preserving the efficacy of current selection methods. Extensive experiments on common benchmarks demonstrate the effectiveness of our method, improving the average accuracy of current selection methods by up to 5.42%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。