利用最大值加法网络的梯度稀疏性，实现更高效的训练。
Exploiting Subgradient Sparsity in Max-Plus Neural Networks
- 基于最大值与求和替代传统运算，天然产生梯度稀疏性。
- 提出稀疏次梯度算法，避免冗余计算，提升优化效率。
- 适合关注模型可解释性与高效训练的研究者。
深度神经网络虽强大，但训练常需密集且昂贵的参数更新。本文采用一种新型最大值加法神经架构，用最大值和求和分别替代传统加法与乘法。该架构在可解释性方面具有潜力，但训练具挑战性。其代数结构天然导致次梯度稀疏：仅对最大值有贡献的神经元影响损失。然而标准反向传播无法利用此稀疏性，造成冗余计算。本文聚焦最小化最差样本损失，将稀疏性传递至优化损失。为此，提出一种显式利用代数稀疏性的稀疏次梯度算法。通过适配最大值加法模型的非光滑特性，方法实现更高效更新并保持理论保证。这为连接代数结构与可扩展学习提供了系统路径。
原文摘要 · Abstract (English)
Deep Neural Networks are powerful tools for solving machine learning problems, but their training often involves dense and costly parameter updates. In this work, we use a novel Max-Plus neural architecture in which classical addition and multiplication are replaced with maximum and summation operations respectively. This is a promising architecture in terms of interpretability, but its training is challenging. A particular feature is that this algebraic structure naturally induces sparsity in the subgradients, as only neurons that contribute to the maximum affect the loss. However, standard backpropagation fails to exploit this sparsity, leading to unnecessary computations. In this work, we focus on the minimization of the worst sample loss which transfers this sparsity to the optimization loss. To address this, we propose a sparse subgradient algorithm that explicitly exploits the algebraic sparsity. By tailoring the optimization procedure to the non-smooth nature of Max-Plus models, our method achieves more efficient updates while retaining theoretical guarantees. This highlights a principled path toward bridging algebraic structure and scalable learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。