通过分层时间抽象，实现语音长序列高效压缩与理解。
FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation
- 设计分层时间抽象模块，逐级压缩语音特征
- 每秒仅需1.67个令牌（压缩97%），保留关键语义
- 适合需要低资源语音处理的长时对话场景
将多模态大语言模型扩展至长时语音面临输入令牌爆炸式增长的瓶颈。现有语音-语言模型直接将高帧率声学特征映射到语言模型输入空间，导致长上下文处理计算开销巨大。与图像或视频不同，语音缺乏空间冗余，极端令牌压缩尤为困难。为此，我们提出FastSLM，一种高效的架构，包含分层时间抽象器（HTA），在多个时间尺度上逐步提炼声学特征。HTA实现了每秒1.67个令牌的极强压缩率（97%减少），同时保留了下游语音-语言理解所需的关键语言信息。实验表明，FastSLM在多种语音-语言任务中表现优异，所需语音令牌数和浮点运算量显著低于现有模型。源代码与模型权重已开源：https://github.com/Lee-junseok1025/FastSLM。
原文摘要 · Abstract (English)
Scaling Multimodal Large Language Models (MLLMs) to long-form speech is bottlenecked by the explosive growth of input tokens. Existing speech-language models project high-frame-rate acoustic features directly into the LLM input space, making long-context processing computationally prohibitive. Unlike images or videos, speech lacks spatial redundancy, making extreme token compression particularly challenging. To address this limitation, we propose FastSLM, a token-efficient architecture featuring the Hierarchical Temporal Abstractor (HTA), which progressively distills acoustic features across multiple temporal scales. HTA achieves an extreme compression rate of 1.67 tokens per second (97% reduction) while preserving essential linguistic information for downstream speech-language understanding. Experimental results demonstrate that FastSLM achieves competitive performance across diverse speech-language tasks while requiring substantially fewer speech tokens and FLOPs than existing speech-language models. The source code and model checkpoints are available at https://github.com/Lee-junseok1025/FastSLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。