通过递归归一化消除数据偏见,让模型持续学习更公平。
Confounder-Free Continual Learning via Recursive Feature Normalization
- 用递归最小二乘法动态更新特征分布,实时抑制混淆因素影响。
- 在持续学习中显著减少因变量变化导致的灾难性遗忘。
- 适用于视觉变换器等架构,适合关注公平性与长期稳定性的场景。
混淆因素是同时影响输入和目标的无关变量,会导致虚假相关和偏差预测。尽管传统模型已有如元数据归一化(MDN)等处理方法,但在持续学习中,如何让模型学习对混淆因素不变的特征表示仍是难题。本文提出递归元数据归一化(R-MDN)层,可嵌入任意深度网络架构(包括视觉变换器),并可在任意模型阶段使用。R-MDN利用递归最小二乘算法,持续更新内部状态以适应数据和混淆变量分布的变化。实验表明,R-MDN在静态学习和持续学习各阶段均能提升不同群体间的预测公平性,有效缓解因混淆因素随时间演变引发的灾难性遗忘。
原文摘要 · Abstract (English)
Confounders are extraneous variables that affect both the input and the target, resulting in spurious correlations and biased predictions. There are recent advances in dealing with or removing confounders in traditional models, such as metadata normalization (MDN), where the distribution of the learned features is adjusted based on the study confounders. However, in the context of continual learning, where a model learns continuously from new data over time without forgetting, learning feature representations that are invariant to confounders remains a significant challenge. To remove their influence from intermediate feature representations, we introduce the Recursive MDN (R-MDN) layer, which can be integrated into any deep learning architecture, including vision transformers, and at any model stage. R-MDN performs statistical regression via the recursive least squares algorithm to maintain and continually update an internal model state with respect to changing distributions of data and confounding variables. Our experiments demonstrate that R-MDN promotes equitable predictions across population groups, both within static learning and across different stages of continual learning, by reducing catastrophic forgetting caused by confounder effects changing over time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。