提出轻量级语音模型推理框架，显著提升效率且不损失性能
FastAdaSP: Multitask-Adapted Efficient Inference for Large Speech Language Model
- 设计加权令牌合并机制，适配语音任务的时序特性
- 实现7倍内存节省和1.83倍解码速度提升，无性能下降
- 适合长序列语音任务，如情感识别与口语问答
本研究探索通过令牌压缩实现多任务语音语言模型（SpeechLM）的高效推理。与视觉或文本不同，语音具有独特的时序依赖性，使得其他模态的高效推理方法难以直接适用。此外，针对长序列和稀疏信号的语音模型高效推理方法仍不充分。为此，我们提出FastAdaSP，一种专为各类语音任务设计的加权令牌合并框架，旨在优化效率与性能之间的权衡。在WavLLM和Qwen-Audio上的实验表明，该方法相较现有基线实现了最先进的效率-性能平衡。具体而言，FastAdaSP在情感识别（ER）和口语问答（SQA）等任务上，实现了7倍内存效率提升和1.83倍解码吞吐量提升，且性能无任何下降。代码将发布于https://github.com/yichen14/FastAdaSP。
原文摘要 · Abstract (English)
In this study, we aim to explore Multitask Speech Language Model (SpeechLM) efficient inference via token reduction. Unlike other modalities such as vision or text, speech has unique temporal dependencies, making previous efficient inference works on other modalities not directly applicable. Furthermore, methods for efficient SpeechLM inference on long sequence and sparse signals remain largely unexplored. Then we propose FastAdaSP, a weighted token merging framework specifically designed for various speech-related tasks to improve the trade-off between efficiency and performance. Experimental results on WavLLM and Qwen-Audio show that our method achieves the state-of-the-art (SOTA) efficiency-performance trade-off compared with other baseline methods. Specifically, FastAdaSP achieved 7x memory efficiency and 1.83x decoding throughput without any degradation on tasks like Emotion Recognition (ER) and Spoken Question Answering (SQA). The code will be available at https://github.com/yichen14/FastAdaSP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。