arXiv:2509.20677cs.LG2025-09被引 2

从谱覆盖角度分析上下文学习稳定性,给出可计算的提示长度推荐方法。

Stability of In-Context Learning: A Spectral Coverage Perspective

  • 基于演示样本的谱覆盖性构造可计算的稳定性判据。
  • 实验显示推荐提示长度始终高于准确率拐点,验证有效性。
  • 方法可校准,适合需要稳定提示设计的研究者使用。

上下文学习(ICL)是大规模语言模型实际部署的关键能力,但其可靠性随提示中示范样本数量变化显著。核心挑战在于,目标概念——示范重采样下的分布稳定性——在大规模下难以直接测量,导致提示长度选择多依赖经验。为此,本文提出一种可计算的充分条件:基于正则化经验二阶矩矩阵的谱分布下尾部作为代理指标。在次高斯表示假设下,推导出非渐近样本量要求(对 $K$ 的下界),以保证该代理事件在指定失败概率下的成立,从而生成保守的提示长度建议。大规模实验表明,所得估计值始终上界于经验准确率拐点,后者仅作为提示长度转变的实际替代而非稳定性定义。在较小独立子集上,基于重采样的分布稳定性测量进一步验证了预期的稳定性解释。最后,通过仅验证的校准步骤,将保守性缩小至约 $1.03$–$1.20$ 倍,同时保持保守排序,为 ICL 提示设计提供实用且可验证的指导。

原文摘要 · Abstract (English)

In-context learning (ICL) is a pivotal capability for the practical deployment of large-scale language models, yet its reliability can vary substantially with the number of demonstrations provided in the prompt. A central obstacle is that the target notion, \emph{distributional stability under demonstration resampling}, is expensive to measure directly at scale, making prompt-length selection largely heuristic. We therefore study a \emph{computable sufficient condition} based on a spectral-coverage proxy: the lower tail of the spectrum of a regularized empirical second-moment matrix formed from demonstration representations. Under sub-Gaussian representation assumptions, we derive a non-asymptotic sample-size requirement (a lower bound on $K$) that guarantees this proxy event with prescribed failure probability, yielding a conservative prompt-length recommendation produced by an observable two-stage estimator. In large-scale experiments, the resulting estimates consistently upper-bound empirical accuracy knee-points, which we treat only as a practical surrogate for the prompt-length transition rather than a definition of stability. On a smaller held-out subset, direct resampling-based distributional stability measurements further validate the intended stability interpretation. Finally, a validation-only calibration step tightens the conservatism (typically to about $1.03$--$1.20\times$) while preserving conservative ordering, providing practical and verifiable guidance for ICL prompt design.

上下文学习提示工程稳定性分析谱方法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。