用随机矩阵理论精简大模型，压缩80%参数仅损失2%准确率。
RMT-KD: Random Matrix Theoretic Causal Knowledge Distillation
- 基于隐层表征的谱特性，保留关键信息方向而非盲目剪枝。
- 在GLUE和CIFAR-10上实现80%参数减少，推理速度提升2.8倍。
- 适合需要高效部署的边缘计算场景，数学基础扎实。
大型深度学习模型如BERT和ResNet虽性能优异，但因规模与算力需求高，难以在边缘设备部署。本文提出RMT-KD，一种基于随机矩阵理论（RMT）的知识蒸馏压缩方法，通过迭代方式逐步减小网络规模。不同于剪枝或启发式秩选择，RMT-KD利用隐藏表示的谱特性识别并保留有信息量的方向。采用逐层RMT因果缩减结合自蒸馏，保障稳定性和准确性。在GLUE和CIFAR-10数据集上，该方法实现最高80%的参数压缩，仅损失2%准确率，推理速度提升2.8倍，功耗近乎减半。结果表明，RMT-KD是一种数学严谨的网络压缩新范式。
原文摘要 · Abstract (English)
Large deep learning models such as BERT and ResNet achieve state-of-the-art performance but are costly to deploy at the edge due to their size and compute demands. We present RMT-KD, a compression method that leverages Random Matrix Theory (RMT) for knowledge distillation to iteratively reduce network size. Instead of pruning or heuristic rank selection, RMT-KD preserves only informative directions identified via the spectral properties of hidden representations. RMT-based causal reduction is applied layer by layer with self-distillation to maintain stability and accuracy. On GLUE and CIFAR-10, RMT-KD achieves up to 80% parameter reduction with only 2% accuracy loss, delivering 2.8x faster inference and nearly halved power consumption. These results establish RMT-KD as a mathematically grounded approach to network distillation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。