BILLNET通过二值化与门控残差设计，实现超低资源视频推理。
BILLNET: A Binarized Conv3D-LSTM Network with Logic-gated residual architecture for hardware-efficient video inference
- 用分组卷积分解3D卷积，降低计算开销。
- 二值化权重与激活，配合门控残差结构，极致压缩内存。
- 多阶段训练支持全量化LSTM，适合边缘设备部署。
长短期记忆网络（LSTM）与3D卷积（Conv3D）在视频任务中表现优异，但需大量内存和计算资源。针对硬件-算法协同设计的高效推理需求，本文提出一种紧凑的二值化Conv3D-LSTM模型BILLNET，适配极低资源硬件。首先，通过两个逐点卷积夹带分组卷积的方式，分解高成本的标准Conv3D；其次，采用MUX-OR门控残差结构，实现权重与激活的二值化；最后，提出多阶段训练策略，支持对LSTM层的完全量化。在Jester数据集上的实验表明，相比现有资源高效的Conv3D模型，该方法在极低内存和计算预算下仍保持高精度。
原文摘要 · Abstract (English)
Long Short-Term Memory (LSTM) and 3D convolution (Conv3D) show impressive results for many video-based applications but require large memory and intensive computing. Motivated by recent works on hardware-algorithmic co-design towards efficient inference, we propose a compact binarized Conv3D-LSTM model architecture called BILLNET, compatible with a highly resource-constrained hardware. Firstly, BILLNET proposes to factorize the costly standard Conv3D by two pointwise convolutions with a grouped convolution in-between. Secondly, BILLNET enables binarized weights and activations via a MUX-OR-gated residual architecture. Finally, to efficiently train BILLNET, we propose a multi-stage training strategy enabling to fully quantize LSTM layers. Results on Jester dataset show that our method can obtain high accuracy with extremely low memory and computational budgets compared to existing Conv3D resource-efficient models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。