提出新方法提升深度网络泛化界紧度，解决权重缩放不变性导致的松散边界问题。
Non-Vacuous Generalization Bounds: Can Rescaling Invariances Help?
- 在不变量提升空间中重定义PAC-Bayes界，消除权重缩放带来的偏差
- 实验证明新方法可获得比传统方法更紧的泛化界，尤其适用于大网络
- 适合关注模型泛化理论、深度学习可解释性的研究者
理解泛化的核心挑战在于获得非平凡的保证，而非仅依赖数据或参数空间的最坏情况复杂度。现有方法中，PAC-Bayes界因其能提供紧致且依赖数据的保证而脱颖而出，即使对大型网络也适用。然而，在ReLU网络中，权重缩放不变性意味着不同的权重分布可能表示相同函数，却导致截然不同的PAC-Bayes复杂度。本文提出在一种不变量提升表示下研究PAC-Bayes界，以解决这一矛盾。该方法不仅具备不变性优势，还能通过数据处理获得更紧的界。同时探讨了基于KL散度的缩放不变性PAC-Bayes界的算法实现问题。
原文摘要 · Abstract (English)
A central challenge in understanding generalization is to obtain non-vacuous guarantees that go beyond worst-case complexity over data or weight space. Among existing approaches, PAC-Bayes bounds stand out as they can provide tight, data-dependent guarantees even for large networks. However, in ReLU networks, rescaling invariances mean that different weight distributions can represent the same function while leading to arbitrarily different PAC-Bayes complexities. We propose to study PAC-Bayes bounds in an invariant, lifted representation that resolves this discrepancy. This paper explores both the guarantees provided by this approach (invariance, tighter bounds via data processing) and the algorithmic aspects of KL-based rescaling-invariant PAC-Bayes bounds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。