arXiv:2509.22580cs.LG2025-09被引 2

现有评估方法会误判模型性能,新方法通过极端序列更真实反映学习能力。

The Lie of the Average: How Class Incremental Learning Evaluation Deceives You?

  • 基于任务相似性动态寻找极端类顺序,提升评估代表性。
  • 实验证明主流方法低估性能波动,平均值偏差可达15%以上。
  • 适合关注模型鲁棒性与真实场景表现的研究者使用。

类增量学习(CIL)要求模型在不遗忘旧知识的前提下持续学习新类别,且需在所有可能的类别序列中保持稳定性能。然而现实场景中类别到达顺序多样且不可预测,不同序列下的模型表现差异显著。主流评估协议仅从少量随机采样的序列中计算均值和方差,理论与实证分析表明该策略无法捕捉完整的性能分布,导致均值估计偏倚,严重低估真实方差。我们提出应准确刻画并估计整个性能分布。为此,引入极端序列概念,并提供理论支持其在可靠评估中的关键作用。同时观察到任务间相似性与模型性能存在一致正相关,可指导极端序列搜索。基于此,提出EDGE(Extreme case-based Distribution & Generalization Evaluation)评估协议,利用任务相似性自适应识别并采样极端序列,更接近真实性能分布。大量实验表明,EDGE有效捕获性能极值,显著提升对分布边界估计的准确性,为模型选择与鲁棒性检验提供可靠依据。代码已开源:https://github.com/AIGNLAI/EDGE。

原文摘要 · Abstract (English)

Class Incremental Learning (CIL) requires models to continuously learn new classes without forgetting previously learned ones, while maintaining stable performance across all possible class sequences. In real-world settings, the order in which classes arrive is diverse and unpredictable, and model performance can vary substantially across different sequences. Yet mainstream evaluation protocols calculate mean and variance from only a small set of randomly sampled sequences. Our theoretical analysis and empirical results demonstrate that this sampling strategy fails to capture the full performance range, resulting in biased mean estimates and a severe underestimation of the true variance in the performance distribution. We therefore contend that a robust CIL evaluation protocol should accurately characterize and estimate the entire performance distribution. To this end, we introduce the concept of extreme sequences and provide theoretical justification for their crucial role in the reliable evaluation of CIL. Moreover, we observe a consistent positive correlation between inter-task similarity and model performance, a relation that can be leveraged to guide the search for extreme sequences. Building on these insights, we propose EDGE (Extreme case-based Distribution & Generalization Evaluation), an evaluation protocol that adaptively identifies and samples extreme class sequences using inter-task similarity, offering a closer approximation of the ground-truth performance distribution. Extensive experiments demonstrate that EDGE effectively captures performance extremes and yields more accurate estimates of distributional boundaries, providing actionable insights for model selection and robustness checking. Our code is available at https://github.com/AIGNLAI/EDGE.

增量学习评估协议性能波动

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。