通过逆转遗忘过程，让模型学会预测更丰富的预训练权重。
Learning from Oblivion: Predicting Knowledge Overflowed Weights via Retrodiction of Forgetting
- 用逐步缩小数据集的微调模拟结构化遗忘，再反向恢复知识。
- 在多个数据集和模型上，新权重使下游任务性能显著提升。
- 适合想提升预训练模型泛化能力的研究者和工程师。
预训练权重已成为现代深度学习的核心，能高效传递知识并提升下游任务表现，尤其在数据稀缺场景下。然而，一个根本问题仍未解决：如何获得包含更多知识的预训练权重？本文提出知识溢出权重（KNOW）预测，通过结构化遗忘及其逆过程合成更丰富的权重。核心洞察是：在逐级缩小的数据集上顺序微调会引发结构化遗忘，该过程可建模并反转，以恢复如同在更大数据集上训练的权重。我们构建了由受控遗忘驱动的权重迁移数据集，并采用元学习有效建模权重预测。具体地，我们的KNowledge-Overflowed Weights Nowcaster（KNOWN）作为超模型，学习权重的一般演化规律，预测具有更好泛化性的增强权重。跨多种数据集与架构的大量实验表明，KNOW预测始终优于朴素微调和简单权重预测，带来更优的下游性能。本工作为重新解读遗忘动态、突破知识迁移极限提供了新视角。代码与预训练模型已公开于https://github.com/jjh6297/KNOW。
原文摘要 · Abstract (English)
Pre-trained weights have become a cornerstone of modern deep learning, enabling efficient knowledge transfer and improving downstream task performance, especially in data-scarce scenarios. However, a fundamental question remains: how can we obtain better pre-trained weights that encapsulate more knowledge beyond the given dataset? In this work, we introduce KNowledge-Overflowed Weights (KNOW) prediction, a novel strategy that leverages structured forgetting and its inversion to synthesize knowledge-enriched weights. Our key insight is that sequential fine-tuning on progressively downsized datasets induces a structured forgetting process, which can be modeled and reversed to recover knowledge as if trained on a larger dataset. We construct a dataset of weight transitions governed by this controlled forgetting and employ meta-learning to model weight prediction effectively. Specifically, our KNowledge-Overflowed Weights Nowcaster (KNOWN) acts as a hyper-model that learns the general evolution of weights and predicts enhanced weights with improved generalization. Extensive experiments across diverse datasets and architectures demonstrate that KNOW prediction consistently outperforms Naive fine-tuning and simple weight prediction, leading to superior downstream performance. Our work provides a new perspective on reinterpreting forgetting dynamics to push the limits of knowledge transfer. The code and pre-trained model are available at https://github.com/jjh6297/KNOW
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。