用对角与二值矩阵分解压缩神经网络权重，节省存储且保持精度。
DiBA: Diagonal and Binary Matrix Approximation for Neural Network Weight Compression
- 将权重矩阵分解为三个对角阵和两个二值阵的乘积，降低计算量。
- 在40个权重矩阵上验证，存储比越低，信号噪声比越高。
- 仅微调对角参数即可适配下游任务，避免二值矩阵重新搜索。
本文提出DiBA（对角与二值矩阵近似），一种紧凑的矩阵分解方法用于神经网络权重压缩。现代网络中线性层、1×1卷积、注意力投影和嵌入层均含稠密权重矩阵。DiBA将矩阵 $A\in\mathbb{R}^{m\times n}$ 近似为 $\widehat A=D_1B_1D_2B_2D_3$，其中 $D_1,D_2,D_3$ 为对角矩阵，$B_1,B_2$ 为0/1二值矩阵，中间维度 $k$ 控制存储与精度的权衡。对于矩阵-向量乘法，DiBA将密集乘法分解为三次逐元素缩放和两次二值混合操作，浮点乘法次数从 $mn$ 降至 $m+k+n$。优化方面，提出DiBA-Greedy，通过闭式最小二乘更新对角因子，结合精确的一比特改进测试更新二值因子。还引入DiBARD（仅重调对角因子的DiBA），将稠密层替换为DiBA因子，冻结二值矩阵，仅在下游数据上微调对角项。该方法在2项组件替换实验中表现优异：在DistilBERT/WikiText上，掩码词预测准确率从0.4447提升至0.5210；在音频频谱变换器上，Speech Commands测试准确率从0.7684升至0.9781，且无需重优化二值矩阵。
原文摘要 · Abstract (English)
In this paper, we propose DiBA (Diagonal and Binary Matrix Approximation), a compact matrix factorization for neural network weight compression. Many components of modern networks, including linear layers, $1\times1$ convolutions, attention projections, and embedding layers, have dense matrix weights. DiBA approximates $A\in\mathbb{R}^{m\times n}$ by $\widehat A=D_1B_1D_2B_2D_3$, where $D_1,D_2,D_3$ are diagonal matrices and $B_1,B_2$ are $0/1$ binary matrices. The intermediate dimension $k$ controls the trade-off between theoretical storage and approximation accuracy. For matrix-vector products, DiBA decomposes dense multiplication into three element-wise scaling operations and two binary mixing operations, reducing the floating-point multiplication count from $mn$ to $m+k+n$. For optimization, we introduce DiBA-Greedy, an alternating solver that combines closed-form least-squares updates for the diagonal factors with exact one-bit improvement tests for the binary factors. We also introduce DiBARD (DiBA with Retuning only Diagonal factors), which replaces dense-matrix layers by DiBA factors, freezes the binary matrices, and retunes only the diagonal entries on downstream data. This preserves compact binary mixing without discrete search during adaptation. On 40 dense weight matrices extracted from public pretrained models, DiBA-Greedy yields consistent SNR improvements as the theoretical storage ratio increases. After DiBA replacement in two component-replacement studies, DiBARD improves DistilBERT/WikiText masked-token accuracy from 0.4447 to 0.5210 and Speech Commands test accuracy for an Audio Spectrogram Transformer from 0.7684 to 0.9781 without reoptimizing the binary factors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。