等变神经网络可能削弱表达能力，增大模型可补偿此缺陷。
Drawback of Enforcing Equivariance and its Compensation via the Lens of Expressive Power
- 通过分析边界超平面与通道向量，揭示等变约束会降低表达力。
- 证明需扩大模型规模才能补偿表达力损失，且放大后假设空间维度下降。
- 适合研究对称性建模与模型泛化性的学者参考。
等变神经网络将数据内在对称性作为归纳偏置，在多个领域表现优异。然而其表达能力的理解仍不充分。本文聚焦两层ReLU网络，研究强制等变约束对表达能力的影响。通过分析边界超平面和通道向量，构造性地证明等变约束可能削弱表达能力。自然地，可通过扩大模型规模来补偿这一缺陷——我们进一步给出了补偿所需的上界。令人惊讶的是，扩大的神经网络架构具有更低的假设空间维度，暗示更强的泛化能力。
原文摘要 · Abstract (English)
Equivariant neural networks encode the intrinsic symmetry of data as an inductive bias, which has achieved impressive performance in wide domains. However, the understanding to their expressive power remains premature. Focusing on 2-layer ReLU networks, this paper investigates the impact of enforcing equivariance constraints on the expressive power. By examining the boundary hyperplanes and the channel vectors, we constructively demonstrate that enforcing equivariance constraints could undermine the expressive power. Naturally, this drawback can be compensated for by enlarging the model size -- we further prove upper bounds on the required enlargement for compensation. Surprisingly, we show that the enlarged neural architectures have reduced hypothesis space dimensionality, implying even better generalizability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。