用最小描述长度原理让神经网络自动选对解，避免过拟合。
A Minimum Description Length Approach to Regularization in Neural Networks
- 用最小描述长度（MDL）平衡模型复杂度与数据拟合度
- 在形式语言任务中，模型能收敛到精确解而非近似解
- 不依赖优化算法，适合追求理论可靠性的研究者
当前最先进的神经网络虽能表达符号化、完美的解，但训练后的模型常停留在近似解。我们发现：使用标准正则化（L1、L2 或无正则）时，即使架构表达能力强，模型也难以收敛到正确解，反而被推向远离理想初始化的方向。相比之下，基于最小描述长度（MDL）原则的正则化方法，在平衡模型复杂度与数据拟合之间提供了理论依据，能优先选择完美解而非近似解，且不依赖具体优化算法。我们提出，与现有方法不同，MDL引入了恰当的归纳偏置，有效抑制过拟合并提升泛化能力。
原文摘要 · Abstract (English)
State-of-the-art neural networks can be trained to become remarkable solutions to many problems. But while these architectures can express symbolic, perfect solutions, trained models often arrive at approximations instead. We show that the choice of regularization method plays a crucial role: when trained on formal languages with standard regularization ($L_1$, $L_2$, or none), expressive architectures not only fail to converge to correct solutions but are actively pushed away from perfect initializations. In contrast, applying the Minimum Description Length (MDL) principle to balance model complexity with data fit provides a theoretically grounded regularization method. Using MDL, perfect solutions are selected over approximations, independently of the optimization algorithm. We propose that unlike existing regularization techniques, MDL introduces the appropriate inductive bias to effectively counteract overfitting and promote generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。