理论分析持续学习中高维岭回归的泛化性能，揭示模型设计对遗忘与迁移的影响。
High-dimensional Asymptotics of Generalization Performance in Continual Ridge Regression
- 基于随机矩阵理论推导高维线性模型下渐近预测风险的精确表达式。
- 发现风险曲线呈现独特现象，模型规格显著影响平均风险、前向/后向迁移效果。
- 适合研究持续学习理论机制的研究者，尤其关注高维数据下的泛化行为。
持续学习旨在应对任务与数据分布随时间变化的实际需求，同时缓解灾难性遗忘问题。尽管持续学习技术取得显著进展，其泛化性能的理论理解仍滞后。本文研究高维线性模型中持续岭回归的理论性质，其中维度与每项任务的样本量成比例。利用随机矩阵理论，我们推导出渐近预测风险的精确表达式，从而刻画持续学习中泛化性能的三项评估指标：平均风险、后向迁移与前向迁移。此外，我们给出了理论风险曲线，展示这些指标在整个持续学习过程中的演变趋势。分析揭示了风险曲线中的若干有趣现象，说明模型设定如何影响泛化性能。仿真研究验证了理论结果的正确性。
原文摘要 · Abstract (English)
Continual learning is motivated by the need to adapt to real-world dynamics in tasks and data distribution while mitigating catastrophic forgetting. Despite significant advances in continual learning techniques, the theoretical understanding of their generalization performance lags behind. This paper examines the theoretical properties of continual ridge regression in high-dimensional linear models, where the dimension is proportional to the sample size in each task. Using random matrix theory, we derive exact expressions of the asymptotic prediction risk, thereby enabling the characterization of three evaluation metrics of generalization performance in continual learning: average risk, backward transfer, and forward transfer. Furthermore, we present the theoretical risk curves to illustrate the trends in these evaluation metrics throughout the continual learning process. Our analysis reveals several intriguing phenomena in the risk curves, demonstrating how model specifications influence the generalization performance. Simulation studies are conducted to validate our theoretical findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。