移除图像数据中的低阶特征会减缓模型学习，但高阶信息可被新架构利用。
Slowing Learning by Erasing Simple Features
- 提出QLEACE方法，精准擦除表示中所有二次可得的概念信息。
- 在前馈网络中，该方法始终减缓学习；复杂架构则能利用注入的高阶信息。
- 近似变体反而像数据增强，某些数据集上比线性擦除更快学习。
已有研究指出，神经网络通常先学习数据分布的低阶矩，再学习高阶相关性。本文提出一种新的闭式概念擦除方法QLEACE，可手术式地从表征中移除关于某一概念的所有二次可用信息。通过与线性擦除（LEACE）及两种近似二次擦除方法对比，探究当图像分类数据集中的低阶统计信息被移除时，网络是否仍能学习。结果发现，尽管LEACE始终减缓学习，二次擦除的效果却因数据集、模型架构和擦除方法而异。使用QLEACE在前馈架构中始终减缓学习，但更复杂的架构能学会利用注入的高阶香农信息。其近似变体避免了信息注入，却意外在部分数据集上表现出数据增强效果，比LEACE更快促进学习。
原文摘要 · Abstract (English)
Prior work suggests that neural networks tend to learn low-order moments of the data distribution first, before moving on to higher-order correlations. In this work, we derive a novel closed-form concept erasure method, QLEACE, which surgically removes all quadratically available information about a concept from a representation. Through comparisons with linear erasure (LEACE) and two approximate forms of quadratic erasure, we explore whether networks can still learn when low-order statistics are removed from image classification datasets. We find that while LEACE consistently slows learning, quadratic erasure can exhibit both positive and negative effects on learning speed depending on the choice of dataset, model architecture, and erasure method. Use of QLEACE consistently slows learning in feedforward architectures, but more sophisticated architectures learn to use injected higher order Shannon information about class labels. Its approximate variants avoid injecting information, but surprisingly act as data augmentation techniques on some datasets, enhancing learning speed compared to LEACE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。