用信息瓶颈原理改进归一化，让模型学得更准、更鲁棒。
IBNorm: Information-Bottleneck Inspired Normalization for Representation Learning
- 基于信息瓶颈设计新归一化，主动保留有用信息
- 在多个大模型上超越现有归一化方法，提升性能
- 适合追求高精度表示学习的开发者和研究者
归一化是深度学习的基础，但现有方法如 BatchNorm、LayerNorm、RMSNorm 以方差为中心，强制均值为零、方差为一，虽稳定训练却未控制表征如何捕捉任务相关信息。我们提出受信息瓶颈启发的归一化（IBNorm），一种简单而强大的方法族。IBNorm 引入有界压缩操作，促使嵌入保留预测信息的同时抑制无关变异，生成更具信息量的表征，同时保持标准归一化的稳定性与兼容性。理论上，我们证明 IBNorm 的信息瓶颈值更高，泛化界更紧。实证上，IBNorm 在大规模语言模型（LLaMA、GPT-2）和视觉模型（ResNet、ViT）中持续优于 BatchNorm、LayerNorm、RMSNorm，互信息分析验证了其更优的信息瓶颈行为。代码将公开发布。
原文摘要 · Abstract (English)
Normalization is fundamental to deep learning, but existing approaches such as BatchNorm, LayerNorm, and RMSNorm are variance-centric by enforcing zero mean and unit variance, stabilizing training without controlling how representations capture task-relevant information. We propose IB-Inspired Normalization (IBNorm), a simple yet powerful family of methods grounded in the Information Bottleneck principle. IBNorm introduces bounded compression operations that encourage embeddings to preserve predictive information while suppressing nuisance variability, yielding more informative representations while retaining the stability and compatibility of standard normalization. Theoretically, we prove that IBNorm achieves a higher IB value and tighter generalization bounds than variance-centric methods. Empirically, IBNorm consistently outperforms BatchNorm, LayerNorm, and RMSNorm across large-scale language models (LLaMA, GPT-2) and vision models (ResNet, ViT), with mutual information analysis confirming superior information bottleneck behavior. Code will be released publicly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。