用非线性张量网络压缩深度模型参数,单层压缩超千倍。
Automatically Differentiable Nonlinear Tensor Networks (ADNTNs) for Exponential Parameter Compression of Deep Neural Networks

- 设计可微的分层张量网络生成器,用少量核心参数生成完整权重
- 在CIFAR-10上实现2000至43万倍的参数压缩,部分模型性能持平基线
- 适合追求模型轻量化且关注结构化压缩方案的研究者
大型深度神经网络存储与部署成本高昂,因推理需处理大量参数。本文研究自动可微的非线性张量网络(ADNTNs),作为替代部分密集层、卷积层和注意力层的紧凑可微权重生成器。ADNTN通过层次化收缩与可学习非线性,将少量可训练张量核映射为完整权重张量,生成层可作为普通线性或卷积算子使用。研究三种多层拓扑:树状张量网络、带边界解纠缠器的增强型树状张量网络,以及类MERA多尺度解码器。相比平坦的砖墙式张量网络,这些分层结构提供对数级深度的张量模式通信与可选横向混合,可在不显著增加存储参数的情况下提升长程结构建模能力。提出统一的前向-伴随公式,支持反向传播计算所有可训练核的预激活伴随与收缩环境梯度,兼容任务损失、重构损失、知识蒸馏、量化感知项、批处理及现代优化器。在CIFAR-10数据集上的典型AlexNet和VGG-16层实验表明,单层参数压缩比达约2,000×至430,000×。多个压缩后的VGG-16模型性能匹配甚至略超原密集基线,而AlexNet在更高约束冗余下出现中等程度性能下降。结果表明,非线性张量网络生成器是实现紧凑模式识别模型的有前景路径,但收缩调度与硬件感知实现仍是实际加速的关键。
原文摘要 · Abstract (English)
Large deep neural networks are costly to store and deploy because inference must move and evaluate many parameters. This paper studies \emph{Automatically Differentiable Nonlinear Tensor Networks} (ADNTNs), compact differentiable weight generators for replacing selected dense, convolutional, and attention layers. An ADNTN maps a small set of trainable tensor cores to a full weight tensor through hierarchical contractions and learnable nonlinearities; the generated layer is then used as an ordinary linear or convolutional operator. We investigate three multilayered topologies: Tree Tensor Networks, augmented Tree Tensor Networks with boundary disentanglers, and MERA-style multi-scale decoders. Compared with flat brick-wall automatically differentiable tensor networks, these hierarchies provide logarithmic-depth communication between tensorised modes and optional lateral mixing, which can improve long-range structure without large increases in stored parameters. We give a unified forward--adjoint formulation showing how reverse-mode automatic differentiation computes pre-activation adjoints and contracted-environment gradients for all trainable cores. The formulation supports task losses, reconstruction losses, distillation, quantisation-aware terms, batching, and modern optimisers. Proof-of-concept experiments on selected AlexNet and VGG-16 layers on CIFAR-10 datasets achieve per-layer parameter-compression ratios from about $2{,}000\times$ to $430{,}000\times$. Several VGG-16 compressed models match or slightly exceed the dense baseline, whereas AlexNet shows moderate degradation under more restrictive redundancy. These results indicate that nonlinear tensor-network generators are a promising structured route to compact pattern-recognition models, while also showing that contraction schedules and hardware-aware implementations remain essential for practical speedups.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。