用语音标记增强语言模型，提升分类任务表现。
A Simple Method to Enhance Pre-trained Language Models with Speech Tokens for Classification
- 通过稀疏特征选择保留关键语音标记，降低融合成本。
- 在论点谬误检测等任务上超越单模态与现有语音模型。
- 适合需要融合语音与文本的多模态分类研究者。
本文提出一种简单方法，可在特定分类任务中对预训练语言模型进行语音信息增强。传统多模态融合面临语音序列远长于文本的问题，而现有语音分词器输出大量高维词汇标记，难以低成本集成。本方法基于多模态词袋表示，采用基于Lasso的特征选择，仅保留对任务重要的音频标记，并通过自监督语言建模适应语言模型，再进行下游任务微调。实验表明，该方法优于单模态模型、更大规模的SpeechLM，以及通过学习表示融合音频的方法。在论点谬误检测、情感计算等任务上验证有效性，即使随机选择音频标记也能提升性能。代码已公开。
原文摘要 · Abstract (English)
This paper presents a simple method that allows to easily enhance textual pre-trained large language models with speech information, when fine-tuned for a specific classification task. A classical issue with the fusion of many embeddings from audio with text is the large length of the audio sequence compared to the text one. Our method benefits from an existing speech tokenizer trained for Audio Speech Recognition that output long sequences of tokens from a large vocabulary, making it difficult to integrate it at low cost in a large language model. By applying a simple lasso-based feature selection on multimodal Bag-of-Words representation, we retain only the most important audio tokens for the task, and adapt the language model to them with a self-supervised language modeling objective, before fine-tuning it on the downstream task. We show this helps to improve the performances compared to an unimodal model, to a bigger SpeechLM or to integrating audio via a learned representation. We demonstrate its effectiveness on Argumentative Fallacy Detection and Classification tasks where audio was previously believed counterproductive, and affective computing tasks on a widely-used dataset. We also provide an in-depth analysis of the method, showing that even a random audio token selection helps enhancing the unimodal model. Our code is available [online](https://github.com/salocinc/EMNLP26SpeechTokLLM/).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。