提出数学框架HOPE，用连续空间解构神经网络内部表示
Hilbert Operator for Progressive Encoding (HOPE): A Mathematical Framework for Deconstructing Learned Representations in Deep Networks
- 将神经元建模为希尔伯特算子，统一剪枝与融合为低秩投影
- 无需数据和超参数，可无偏评估不同层的结构压缩效果
- 适合研究模型压缩、知识解构的理论工作者和架构设计者
深度神经网络编码复杂表征，但解析其内部知识仍具挑战。鉴于学习与压缩之间的关联，网络压缩为分析此类知识提供了新视角。然而，标准压缩启发式常受尺度对称性和架构偏见影响。为此，我们提出希尔伯特算子渐进编码（HOPE）——一种数学框架，用于逐步解构训练后网络权重中的表征。HOPE将网络压缩从离散域转移至连续函数的希尔伯特空间。通过将单个神经元建模为一阶希尔伯特-施密特算子，HOPE将剪枝与神经元合并统一为低秩子空间投影。在此基础上，HOPE引入宏观块剔除机制，将整个残差路径等多层结构纳入同一度量体系。该统一方法实现跨层、跨类型、跨规模的无偏架构决策。HOPE为无需数据且无需超参数的框架。我们在模型压缩与微调中展示概念验证实验，凸显该理论的实际潜力。
原文摘要 · Abstract (English)
Deep neural networks encode complex representations, but deconstructing this internal knowledge remains a challenge. Given the link between learning and compression, network compression offers a promising lens to analyze this knowledge. However, standard compression heuristics often suffer from scale symmetries and architectural biases. To resolve these, we introduce Hilbert Operator for Progressive Encoding (HOPE), a mathematical framework to gradually deconstruct the representations in trained network weights. HOPE shifts network compression from the discrete domain into a Hilbert space of continuous functions. By modeling individual neurons as rank-1 Hilbert-Schmidt operators, HOPE unifies pruning and neuron merging as low-rank subspace projection. Extending this formulation, HOPE introduces macro block eviction to encompass multi-layer structures like entire residual pathways under the same unified metric. This unified approach enables unbiased architectural decisions across layers with different types and sizes. HOPE is a data-free and hyperparameter-free framework. We present proof-of-concept experiments in model compression and fine-tuning to highlight the practical potential of our theory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。