arXiv:2605.27469cs.LGcs.AI2026-05

提出轻量级架构驱动的逻辑偏移预测方法,高效评估持续学习模型性能。

Architecture-driven Shift: towards a lightweight selector for capturing the trends of logit shift

论文配图:Architecture-driven Shift: towards a lightweight selector for capturing the trends of logit shift
图 1 · 摘自论文原文
  • 分解逻辑偏移为架构与数据依赖,构建可计算的架构驱动偏移(ADS)
  • ADS与逻辑偏移呈强单调相关(最弱斯皮尔曼相关系数0.731)
  • 仅需少量样本即可预测校准误差,适合大规模模型筛选

持续学习(CL)利用深度预训练网络的潜力,但如何选择在可塑性与稳定性间平衡更好的预训练模型仍具挑战。逻辑偏移作为自然代理指标反映CL场景下的输出变化,但其计算成本高昂。现有理论分析因假设各层宽度一致,忽略真实模型的结构异质性,难以提供高效替代方案。本文提出架构驱动偏移(ADS),将逻辑偏移解耦为架构依赖与数据依赖,揭示二者结合能有效捕捉逻辑偏移趋势。对于在先验任务上优化良好的模型,更高的ADS值对应当前任务训练后更大的逻辑偏移,该关系由三个机制支撑:(1)权重梯度谱范数随层宽缩放,(2)新任务优化路径长度,(3)宽网络中的渐近任务冲突。在超过175种异构架构上的实验证明,ADS与逻辑偏移存在强单调相关性(最弱斯皮尔曼相关系数$r_s=0.731$)。实际应用中,ADS可作为预期校准误差的轻量代理,已在三个数据集六种场景下验证有效性。

原文摘要 · Abstract (English)

Continual Learning (CL) is a practical paradigm to utilize power of deep pre-trained neural networks, but which pre-trained model has a better ability to balance ``Plasticity-Stability", deserving to be chosen? The logit shift serves as a natural proxy because it represents the logit shift in CL scenarios. However, obtaining the logit shift requires huge computational cost, which hinders large-scale model selection. Existing theoretical analyses fail to offer an efficient alternative because of the assumption of uniform hidden layer widths, which ignores the structural heterogeneity (variable width and depth) of real-world architectures. This raises a critical question: what theoretically relationship can be identified between heterogeneous architecture and logit shift on prior tasks (that the model has been trained on)? To answer the question, we decouple logit shift into architecture dependency and data dependency to establish our framework, which reveals that the combination of two dependency, defined as Architecture-driven Shift (ADS), that can capture the logit shift tendency well computable with few data samples. Specifically, for a well-optimized model on prior tasks, higher ADS is associated with a larger logit shift after training on the current task, which derived based on three mechanistic components: (1) spectral norm scaling of weight matrix gradients with layer width, (2) the optimization path length of the new task, and (3) the asymptotic task conflict in wide networks. Extensive empirical results across more than 175 diverse architectures demonstrate a strong monotonic correlation (the weakest Spearman's $r_s=0.731$) between ADS and logit shift. Practically, we demonstrate that ADS can serve as a lightweight proxy of the expected calibration error, which is a widely used metric for reliable CL model selection, on three datasets across six scenarios.

持续学习模型选择架构分析轻量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。