用哈达玛矩阵实现二值化正交RNN，让边缘设备也能跑高效长序列模型。
HadamRNN: Binary and Sparse Ternary Orthogonal RNNs
- 基于哈达玛矩阵构造二值/稀疏三值正交权重，解决RNN权重量化难题
- 在1000步拷贝任务中表现优异，性能接近全精度模型
- 适合资源受限场景下的长序列建模，如物联网设备部署
神经网络中的二值化和稀疏三值权重能实现更快的计算和更轻的表示，便于在计算能力有限的边缘设备上运行。然而，传统RNN对循环权重变化极为敏感，使得其权重的二值化和三值化始终难以实现。迄今为止，尚无方法成功实现传统RNN权重的二值化或三值化。本文提出一种新方法，利用哈达玛矩阵的性质参数化一类二值及稀疏三值正交矩阵，从而实现具有二值和稀疏三值循环权重的正交RNN（ORNN）训练，构建出特定类别的二值与稀疏三值传统RNN。所提出的ORNN模型称为HadamRNN和Block-HadamRNN，分别在拷贝任务、置换/顺序MNIST、IMDB数据集、两个GLUE基准以及两个IoT基准上进行了评估。尽管经过二值化或稀疏三值化处理，这些RNN仍保持与当前最优全精度模型相当的性能水平，证明了该方法的有效性。值得注意的是，本方法是首个能够处理超过1000个时间步拷贝任务的二值循环权重方案。
原文摘要 · Abstract (English)
Binary and sparse ternary weights in neural networks enable faster computations and lighter representations, facilitating their use on edge devices with limited computational power. Meanwhile, vanilla RNNs are highly sensitive to changes in their recurrent weights, making the binarization and ternarization of these weights inherently challenging. To date, no method has successfully achieved binarization or ternarization of vanilla RNN weights. We present a new approach leveraging the properties of Hadamard matrices to parameterize a subset of binary and sparse ternary orthogonal matrices. This method enables the training of orthogonal RNNs (ORNNs) with binary and sparse ternary recurrent weights, effectively creating a specific class of binary and sparse ternary vanilla RNNs. The resulting ORNNs, called HadamRNN and Block-HadamRNN, are evaluated on benchmarks such as the copy task, permuted and sequential MNIST tasks, the IMDB dataset, two GLUE benchmarks, and two IoT benchmarks. Despite binarization or sparse ternarization, these RNNs maintain performance levels comparable to state-of-the-art full-precision models, highlighting the effectiveness of our approach. Notably, our approach is the first solution with binary recurrent weights capable of tackling the copy task over 1000 timesteps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。