揭示对比学习在数据不均衡下的训练动态，提出剪枝修复方案
Theoretical Analysis of Contrastive Learning under Imbalanced Data: From Training Dynamics to a Pruning Solution
- 分析变压器编码器在不均衡数据下的神经元权重三阶段演化
- 发现少数类特征降低表征能力，需更复杂结构且难分清真实信号与噪声
- 剪枝可恢复性能，适合处理现实场景中的数据偏斜问题
对比学习已成为学习泛化表示的强大框架，但其理论理解仍不充分，尤其是在现实应用中普遍存在的数据不均衡情况下。这种不平衡会损害表示质量并引发模型偏差，但对其影响的严谨刻画尚缺。本文构建了针对基于变压器编码器的对比学习在不均衡数据下的训练动态理论框架。结果表明，神经元权重在训练中经历三个不同阶段，多数特征、少数特征与噪声分别呈现不同演化规律。进一步显示，少数特征会降低表征容量，增加对更复杂架构的需求，并阻碍真实特征与噪声的分离。受此神经元层面行为启发，我们证明剪枝可恢复由不平衡导致的性能下降，提升特征分离效果，兼具理论洞见与实践指导意义。主要理论发现通过数值实验得到验证。
原文摘要 · Abstract (English)
Contrastive learning has emerged as a powerful framework for learning generalizable representations, yet its theoretical understanding remains limited, particularly under imbalanced data distributions that are prevalent in real-world applications. Such an imbalance can degrade representation quality and induce biased model behavior, yet a rigorous characterization of these effects is lacking. In this work, we develop a theoretical framework to analyze the training dynamics of contrastive learning with Transformer-based encoders under imbalanced data. Our results reveal that neuron weights evolve through three distinct stages of training, with different dynamics for majority features, minority features, and noise. We further show that minority features reduce representational capacity, increase the need for more complex architectures, and hinder the separation of ground-truth features from noise. Inspired by these neuron-level behaviors, we show that pruning restores performance degraded by imbalance and enhances feature separation, offering both conceptual insights and practical guidance. Major theoretical findings are validated through numerical experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。