用‘意外度’动态调整模型预测，提升大模型少样本学习的准确性
Surprise Calibration for Better In-Context Learning
- 基于贝叶斯推理框架，用‘意外度’捕捉类别先验变化
- 在多个NLP基准上优于传统校准方法，性能更稳定
- 适合需要灵活适应不同上下文的少样本学习场景
上下文学习（ICL）已成为大语言模型任务适配的强大范式，模型通过少量示例推断任务结构。然而，ICL易受先验知识和上下文示例带来的偏差影响，导致性能下降。现有偏差校准方法通常对所有输入使用固定的类别先验，难以适应动态的ICL场景。为此，本文采用隐式顺序贝叶斯推理框架解释ICL，识别‘意外度’作为类别先验转移的有用信号，并提出新方法——惊喜校准（Surprise Calibration, SC）。SC利用‘意外度’捕捉类别先验的时间动态性，提供更自适应且计算高效的上下文学习解决方案。我们在多个自然语言处理基准任务上实证表明，SC显著优于现有偏差校准技术。
原文摘要 · Abstract (English)
In-context learning (ICL) has emerged as a powerful paradigm for task adaptation in large language models (LLMs), where models infer underlying task structures from a few demonstrations. However, ICL remains susceptible to biases that arise from prior knowledge and contextual demonstrations, which can degrade the performance of LLMs. Existing bias calibration methods typically apply fixed class priors across all inputs, limiting their efficacy in dynamic ICL settings where the context for each query differs. To address these limitations, we adopt implicit sequential Bayesian inference as a framework for interpreting ICL, identify "surprise" as an informative signal for class prior shift, and introduce a novel method--Surprise Calibration (SC). SC leverages the notion of surprise to capture the temporal dynamics of class priors, providing a more adaptive and computationally efficient solution for in-context learning. We empirically demonstrate the superiority of SC over existing bias calibration techniques across a range of benchmark natural language processing tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。