arXiv:2607.05457cs.LGcs.AI2026-07

基于可控可观测性测试,实现神经网络状态压缩与精度保持。

Empirical Minimal-Realisation Compression of Deep Neural Networks via Controllability-Observability Tests

  • 通过数据驱动构建可控可观测性矩阵,评估各层状态冗余。
  • 在MNIST和CIFAR-10上分别实现72.95%和70.94%的状态压缩。
  • 适合追求模型轻量化且需保留精度的工程部署场景。

深度神经网络通常存在大量隐状态冗余,但现有压缩方法多直接作用于权重、神经元或量化表示,未显式刻画内部状态的动力学角色。本文提出一种基于可控性-可观测性的经验状态阶数简化框架,将训练好的网络视为深度索引的非线性动力系统,从隐藏状态快照与输出雅可比矩阵中构建数据驱动的可达性、可观测性和平衡格拉米安。由此得到的A/B/C测试可估计每层的可达、可观测及联合可达可观测秩。这些秩不仅作为隐状态冗余的诊断指标,还可直接用于确定压缩后的层宽。在MNIST和CIFAR-10上的实验表明:在四层SiLU DNN上,状态阶数由1024降至277,实现72.95%状态压缩和73.48%参数压缩,准确率从96.60%降至95.45%;在更大的SiLU DNN上,状态阶数由4608降至1339,实现70.94%状态压缩和83.09%参数压缩,准确率从54.45%降至54.44%,并使CUDA推理延迟降低约3倍。结果表明,平衡的可达-可观测秩为设计紧凑架构提供了原则性依据,几乎无精度损失。

原文摘要 · Abstract (English)

Deep neural networks often contain substantial hidden-state redundancy, but most compression methods operate directly on weights, neurons, or quantised representations without explicitly characterising the dynamical role of internal states. This paper proposes a controllability-observability framework for empirical state-order reduction of deep neural networks. By viewing a trained network as a depth-indexed nonlinear dynamical system, we construct data-driven reachability, observability, and balanced Gramians from hidden-state snapshots and output Jacobians. The resulting A/B/C tests estimate layer-wise reachable, observable, and jointly reachable--observable ranks. These ranks are then used not only as diagnostic measures of hidden-state redundancy, but also as actual compressed layer widths for realised reduced networks. Experiments on MNIST and CIFAR-10 compare the proposed balanced realisation against projection-based reduction, unstructured pruning, structured pruning, low-rank SVD, dynamic INT8 quantisation, and linear baselines. On MNIST, a four-layer SiLU DNN is reduced from state order 1024 to 277, giving 72.95% state compression and 73.48% parameter compression, while maintaining 95.45% accuracy compared with 96.60% for the full model. On CIFAR-10, a larger SiLU DNN is reduced from state order 4608 to 1339, giving 70.94% state compression and 83.09% parameter compression, while preserving accuracy from 54.45% to 54.44% and reducing CUDA inference latency by approximately 3X. The results show that balanced reachable-observable ranks provide a principled empirical minimal-realisation criterion for designing compact neural architectures with little or no loss in accuracy.

模型压缩神经网络状态压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。