通过渐进冻结层，避免表示坍缩，提升自监督视觉表征学习效果。
LayerLock: Non-collapsing Representation Learning with Progressive Freezing
- 按深度顺序渐进冻结ViT层，从像素预测平滑过渡到潜在特征预测。
- 在4DS感知套件上，40亿参数模型性能超越非潜在掩码预测方法。
- 适用于大规模模型，无需复杂设计即可稳定训练，适合视觉表征研究者。
我们提出LayerLock，一种简单有效的自监督视觉表征学习方法，通过渐进式层冻结实现从像素到潜在表示的平稳过渡。首先观察到，在视频掩码自编码（MAE）模型训练中，ViT各层按深度顺序收敛：浅层先收敛，深层后收敛。基于此现象，我们设计了一个显式调度策略，逐步冻结模型，从而加速标准MAE训练。此外，该调度还可用于一种简单且可扩展的潜在表示预测方法，有效避免了“表示坍缩”问题。我们将LayerLock应用于最大达40亿参数的模型，在4DS感知套件上取得了优于非潜在掩码预测的结果。
原文摘要 · Abstract (English)
We introduce LayerLock, a simple yet effective approach for self-supervised visual representation learning, that gradually transitions from pixel to latent prediction through progressive layer freezing. First, we make the observation that during training of video masked-autoencoding (MAE) models, ViT layers converge in the order of their depth: shallower layers converge early, deeper layers converge late. We then show that this observation can be exploited to accelerate standard MAE by progressively freezing the model according to an explicit schedule, throughout training. Furthermore, this same schedule can be used in a simple and scalable approach to latent prediction that does not suffer from "representation collapse". We apply our proposed approach, LayerLock, to large models of up to 4B parameters with results surpassing those of non-latent masked prediction on the 4DS perception suite.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。