发现稀疏率降低能有效提升模型泛化能力，为设计更优模型提供理论依据。
An In-depth Investigation of Sparse Rate Reduction in Transformer-like Models
- 通过分析CRATE模型各层行为，提出多种SRR实现方法。
- 实验验证SRR与泛化能力正相关，优于路径范数等传统度量。
- 将SRR作为正则项可提升图像分类模型性能，适合模型优化研究者。
深度神经网络长期被批评为黑箱。为揭示现代神经架构的内部机制，近期工作提出基于信息论的目标函数稀疏率降低（SRR），并将其优化过程解释为类似Transformer的编码率降低变压器（CRATE）。然而该研究主要关注基础实现，实际优化中是否优化了此目标及其与泛化的关系尚不明确。本文通过分析CRATE的逐层行为，从理论和实证角度推导出不同实现方式。为揭示SRR对泛化的预测能力，我们收集了由不同实现方式和超参数生成的一系列模型变体，并基于其与泛化性能的相关性评估SRR作为复杂度度量的效果。令人惊讶的是，SRR表现出正相关系数，且优于路径范数、尖锐度等基线度量。此外，我们在基准图像分类数据集上证明，以SRR作为正则化项可提升模型泛化能力。本研究希望为利用SRR设计有原则的模型及研究其泛化能力提供启示。
原文摘要 · Abstract (English)
Deep neural networks have long been criticized for being black-box. To unveil the inner workings of modern neural architectures, a recent work \cite{yu2024white} proposed an information-theoretic objective function called Sparse Rate Reduction (SRR) and interpreted its unrolled optimization as a Transformer-like model called Coding Rate Reduction Transformer (CRATE). However, the focus of the study was primarily on the basic implementation, and whether this objective is optimized in practice and its causal relationship to generalization remain elusive. Going beyond this study, we derive different implementations by analyzing layer-wise behaviors of CRATE, both theoretically and empirically. To reveal the predictive power of SRR on generalization, we collect a set of model variants induced by varied implementations and hyperparameters and evaluate SRR as a complexity measure based on its correlation with generalization. Surprisingly, we find out that SRR has a positive correlation coefficient and outperforms other baseline measures, such as path-norm and sharpness-based ones. Furthermore, we show that generalization can be improved using SRR as regularization on benchmark image classification datasets. We hope this paper can shed light on leveraging SRR to design principled models and study their generalization ability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。