将神经网络权重转为高阶张量，提升压缩与可解释性
Tensorization is a powerful but underexplored tool for compression and interpretability of neural networks
- 把权重矩阵转成高阶张量，用低秩张量分解压缩
- 保留键特征路径，展现跨层特征演化规律
- 适合关注模型压缩和机制可解释性的研究者
张量化神经网络是指将部分或全部密集权重矩阵重塑为高阶张量，并使用低秩张量网络分解进行近似。该技术在大规模神经网络的模型压缩方面展现出潜力。尽管已有令人鼓舞的实证结果，张量化神经网络（TNNs）在主流深度学习中仍应用不足。本文从潜力与局限两方面阐述了TNNs的价值。我们认为，TNNs是强大却未被充分探索的深度学习框架，应获得更多工程与理论界关注。除压缩外，其作为具有独特缩放特性的灵活架构，还提升了可解释性。关键特征是引入了键索引（bond indices），形成传统网络中不存在的新隐空间，有助于理解特征在各层间的演变，推动机制可解释性目标。最后，我们提出若干研究方向，旨在克服当前规模化与集成到现代深度学习工作流中的实际障碍。
原文摘要 · Abstract (English)
Tensorizing a neural network involves reshaping some or all of its dense weight matrices into higher-order tensors and approximating them using low-rank tensor network decompositions. This technique has shown promise as a model compression strategy for large-scale neural networks. However, despite encouraging empirical results, tensorized neural networks (TNNs) remain underutilized in mainstream deep learning. In this position paper, we offer a perspective on both the potential and current limitations of TNNs. We argue that TNNs represent a powerful yet underexplored framework for deep learning--one that deserves greater attention from both engineering and theoretical communities. Beyond compression, we highlight the value of TNNs as a flexible class of architectures with distinctive scaling properties and increased interpretability. A central feature of TNNs is the presence of bond indices, which introduce new latent spaces not found in conventional networks. These internal representations may provide deeper insight into the evolution of features across layers, potentially advancing the goals of mechanistic interpretability. We conclude by outlining several key research directions aimed at overcoming the practical barriers to scaling and adopting TNNs in modern deep learning workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。