低容量神经网络在复杂任务中如何平衡泛化、鲁棒性和可解释性
Understanding Generalization, Robustness, and Interpretability in Low-Capacity Neural Networks
- 用渐进困难的MNIST二分类任务控制实验，研究模型容量与性能关系
- 模型容量需随任务复杂度线性增加，95%稀疏度下仍保持高精度
- 稀疏子网络能保留原始模型的核心推理过程，适合小规模部署
尽管现代深度学习依赖大规模过参数化模型，但低容量网络中容量、稀疏性与鲁棒性的基本关系仍是重要研究方向。我们通过从MNIST数据集构建一系列视觉难度递增的二分类任务（如0和1对比4和9），建立可控实验框架。实验揭示三个核心发现：第一，成功泛化所需的最小模型容量与任务复杂度呈正比；第二，训练后的网络对极端幅度剪枝（高达95%稀疏度）具有鲁棒性，表明存在稀疏且高性能的子网络；第三，过参数化显著提升对输入扰动的鲁棒性。通过显著性图的可解释性分析进一步证实，这些稀疏子网络保留了原密集模型的核心推理过程。本工作为简单神经网络的基本权衡提供了清晰的实证展示。
原文摘要 · Abstract (English)
Although modern deep learning often relies on massive over-parameterized models, the fundamental interplay between capacity, sparsity, and robustness in low-capacity networks remains a vital area of study. We introduce a controlled framework to investigate these properties by creating a suite of binary classification tasks from the MNIST dataset with increasing visual difficulty (e.g., 0 and 1 vs. 4 and 9). Our experiments reveal three core findings. First, the minimum model capacity required for successful generalization scales directly with task complexity. Second, these trained networks are robust to extreme magnitude pruning (up to 95% sparsity), revealing the existence of sparse, high-performing subnetworks. Third, we show that over-parameterization provides a significant advantage in robustness against input corruption. Interpretability analysis via saliency maps further confirms that these identified sparse subnetworks preserve the core reasoning process of the original dense models. This work provides a clear, empirical demonstration of the foundational trade-offs governing simple neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。