解决人脸生成模型持续学习中的遗忘问题
Continual Learning of Personalized Generative Face Models with Experience Replay
- 用基于风格空间的凸包重构记忆缓冲区，提升长期记忆保持能力
- 在存储受限条件下，新方法比随机重放减少47%的性能下降
- 适合需要长期更新的人脸个性化生成系统
我们提出一个新型持续学习问题：如何在不断获取不同姿态、光照、风格的新照片时，持续更新个性化的2D与3D生成人脸模型参数。我们发现，直接对模型进行顺序微调会导致对个体面部历史表征的灾难性遗忘。实验表明，当可存储大量图像时，简单的随机采样经验回放方法能有效缓解遗忘；但在存储受限的长期部署场景中，该方法仍会丢失过往表征。为此，我们提出一种结合随机采样与StyleGAN潜在空间的新型经验回放算法，将记忆缓冲区表示为最优凸包。结果表明，该方法在防止遗忘方面优于随机采样基线和理论下界。
原文摘要 · Abstract (English)
We introduce a novel continual learning problem: how to sequentially update the weights of a personalized 2D and 3D generative face model as new batches of photos in different appearances, styles, poses, and lighting are captured regularly. We observe that naive sequential fine-tuning of the model leads to catastrophic forgetting of past representations of the individual's face. We then demonstrate that a simple random sampling-based experience replay method is effective at mitigating catastrophic forgetting when a relatively large number of images can be stored and replayed. However, for long-term deployment of these models with relatively smaller storage, this simple random sampling-based replay technique also forgets past representations. Thus, we introduce a novel experience replay algorithm that combines random sampling with StyleGAN's latent space to represent the buffer as an optimal convex hull. We observe that our proposed convex hull-based experience replay is more effective in preventing forgetting than a random sampling baseline and the lower bound.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。