提出三值量化与图谱编码加速纠错变压器,大幅降低资源消耗。
Accelerating Error Correction Code Transformers
- 采用三值权重量化实现无乘法线性层,减少计算开销。
- 通过码感知多头机制优化自注意力,降低复杂度;结合图谱分解增强表示。
- 在保持性能前提下压缩90%模型,能耗降低224倍,适合嵌入式部署。
纠错码(ECC)在通信系统中对可靠信息传输至关重要。Choukroun & Wolf(2022b)提出的纠错码变换器(ECCT)在多种信道和码族上表现优异,但其高计算与内存需求限制了实际应用,尤其在紧凑神经网络中,现有极低精度量化方法常导致性能下降。本文提出一种新型加速方法:首先设计针对ECCT的三值权重量化,实现无乘法线性层;其次提出码感知多头处理的优化自注意力机制,降低计算复杂度;最后通过泰纳图(Tanner graph)特征分解提供位置编码,增强图连通性表征。该方法在匹配或超越原ECCT性能的同时,显著降低能耗、内存占用与计算复杂度,在现代硬件上实现至少224倍的算术操作能耗缩减,模型压缩率达90%,使基于变压器的纠错更贴近资源受限环境的实际部署。
原文摘要 · Abstract (English)
Error correction codes (ECC) are crucial for ensuring reliable information transmission in communication systems. Choukroun & Wolf (2022b) recently introduced the Error Correction Code Transformer (ECCT), which has demonstrated promising performance across various transmission channels and families of codes. However, its high computational and memory demands limit its practical applications compared to traditional decoding algorithms. Achieving effective quantization of the ECCT presents significant challenges due to its inherently small architecture, since existing, very low-precision quantization techniques often lead to performance degradation in compact neural networks. In this paper, we introduce a novel acceleration method for transformer-based decoders. We first propose a ternary weight quantization method specifically designed for the ECCT, inducing a decoder with multiplication-free linear layers. We present an optimized self-attention mechanism to reduce computational complexity via codeaware multi-heads processing. Finally, we provide positional encoding via the Tanner graph eigendecomposition, enabling a richer representation of the graph connectivity. The approach not only matches or surpasses ECCT's performance but also significantly reduces energy consumption, memory footprint, and computational complexity. Our method brings transformer-based error correction closer to practical implementation in resource-constrained environments, achieving a 90% compression ratio and reducing arithmetic operation energy consumption by at least 224 times on modern hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。