用廉价操作生成冗余状态，让RNN更省资源却保持性能
GhostRNN: Reducing State Redundancy in RNN with Cheap Operations
- 先生成少量核心状态，再用低成本操作生成其余状态
- 在关键词识别和语音增强任务中降低40%内存占用
- 适合部署在算力有限的设备上的高效语音模型
能够建模长距离依赖的循环神经网络（RNN）广泛应用于各类语音任务，如关键词检测（KWS）和语音增强（SE）。由于低资源设备在功耗和内存上的限制，亟需高效的RNN模型以支持实际应用。本文提出一种新型高效RNN架构GhostRNN，通过廉价操作减少隐藏状态冗余。我们观察到，在训练后的RNN模型中，部分隐藏状态维度与其他维度高度相似，表明存在冗余。为此，我们首先生成少量基础状态，再通过廉价操作基于这些基础状态生成“幽灵状态”。在KWS和SE任务上的实验表明，所提GhostRNN显著降低了内存使用量（约40%）和计算开销，同时保持了相近的性能表现。
原文摘要 · Abstract (English)
Recurrent neural network (RNNs) that are capable of modeling long-distance dependencies are widely used in various speech tasks, eg., keyword spotting (KWS) and speech enhancement (SE). Due to the limitation of power and memory in low-resource devices, efficient RNN models are urgently required for real-world applications. In this paper, we propose an efficient RNN architecture, GhostRNN, which reduces hidden state redundancy with cheap operations. In particular, we observe that partial dimensions of hidden states are similar to the others in trained RNN models, suggesting that redundancy exists in specific RNNs. To reduce the redundancy and hence computational cost, we propose to first generate a few intrinsic states, and then apply cheap operations to produce ghost states based on the intrinsic states. Experiments on KWS and SE tasks demonstrate that the proposed GhostRNN significantly reduces the memory usage (~40%) and computation cost while keeping performance similar.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。