通过贝叶斯框架实现任务间知识传递，让模型既能保留旧知识又可灵活适应新任务。
Low-Rank Filtering and Smoothing for Sequential Deep Learning
- 将参数视为非线性高斯状态空间，用低秩近似实现高效滤波与平滑
- 支持在不同层控制适应程度，且允许后续任务反向影响前期模型
- 无需访问原始数据即可共享知识，适合隐私敏感场景
顺序学习多个任务时，神经网络需在保留旧知识与适应新任务之间取得平衡。常规参数正则化方法通常不考虑任务间关系，且仅允许信息单向流向未来任务。本文提出一种贝叶斯框架，将网络参数视为非线性高斯模型的状态空间，实现两大能力：(1) 可靠编码任务间先验知识，例如控制哪些层在任务间进行调整；(2) 创新性应用贝叶斯平滑，使特定任务模型可融合后期学习到的模型知识，而无需直接访问其数据，这对隐私敏感应用至关重要。上述能力依赖于高效的滤波与平滑操作，为此我们提出了拉普拉斯近似中精度矩阵的对角加低秩逼近（LR-LGF）。实验表明，LR-LGF 具有高效率，且所解锁的能力显著提升性能。
原文摘要 · Abstract (English)
Learning multiple tasks sequentially requires neural networks to balance retaining knowledge, yet being flexible enough to adapt to new tasks. Regularizing network parameters is a common approach, but it rarely incorporates prior knowledge about task relationships, and limits information flow to future tasks only. We propose a Bayesian framework that treats the network's parameters as the state space of a nonlinear Gaussian model, unlocking two key capabilities: (1) A principled way to encode domain knowledge about task relationships, allowing, e.g., control over which layers should adapt between tasks. (2) A novel application of Bayesian smoothing, allowing task-specific models to also incorporate knowledge from models learned later. This does not require direct access to their data, which is crucial, e.g., for privacy-critical applications. These capabilities rely on efficient filtering and smoothing operations, for which we propose diagonal plus low-rank approximations of the precision matrix in the Laplace approximation (LR-LGF). Empirical results demonstrate the efficiency of LR-LGF and the benefits of the unlocked capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。