用新型注意力机制提升语音语言识别准确率
On the use of Performer and Agent Attention for Spoken Language Identification
- 引入Performer和Agent注意力替代传统自注意力
- Performer在三个数据集上均优于自注意力，性能更优
- Agent注意力计算更省资源，适合部署场景
语音语言识别（LID）通常采用自监督预训练模型提取语音表征，再微调用于任务识别。当前先进方法使用基于注意力的统计池化层，聚合嵌入向量在时间维度上的上下文信息。本文探索了近期提出的Performer和Agent注意力机制与统计池化层的结合效果。在VoxPopuli、FLEURS和VoxLingua三个数据集上进行实验，对比其与原始自注意力的表现。结果表明，Performer注意力在所有数据集上均优于自注意力；Agent注意力表现相当或略优，且计算成本更低。
原文摘要 · Abstract (English)
One of the methods for language Identification (LID) involves deriving speech representation from pre-trained models using self-supervised learning, followed by fine-tuning the model for the LID task. State-of-the-art approaches for LID use an attention-based statistical pooling layer to facilitate the aggregation of contextual information across time frames of the embedding vectors extracted from the pre-trained model. In this paper, we delve into exploring recently proposed attention mechanisms, namely performer and agent-attention, in conjunction with the statistical pooling layer. The LID experiments are performed on three datasets: VoxPopuli, FLEURS, and VoxLingua. We compare their performance against vanilla self-attention. Our findings suggest that performer-attention outperforms self-attention and agent-attention exhibits comparable or occasionally superior performance to self-attention, while also being computationally less expensive.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。