用轻量方法让大模型具备说话人验证能力，效果接近专业系统。
Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation
- 提出无模型依赖的评分协议，统一评估各类语音大模型的说话人识别能力。
- 在VoxCeleb1-E数据集上，改进后模型达到1.03%的错误率，接近专用系统。
- 仅训练低秩适配器，即可让小模型实现说话人验证，保留自然语言交互。
语音感知型大语言模型（LLM）虽可接收语音输入，但其训练目标多聚焦于语言内容或特定领域（如情感、说话人性别），尚未明确其是否编码说话人身份。本文首先提出一种模型无关的评分协议，通过判断‘是/否’标记的概率置信度或对数似然比，为仅提供API或开放权重的模型生成连续验证得分。基于该协议，我们评估了近期语音感知型LLM，发现其说话人区分能力较弱（在VoxCeleb1上等错误率EER > 20%）。其次，提出一种轻量级增强方法：通过一个可学习的投影层注入冻结的ECAPA-TDNN说话人嵌入，仅训练LoRA适配器，使模型具备说话人验证能力。在TinyLLaMA-1.1B上，所提ECAPA-LLM在VoxCeleb1-E上实现1.03% EER，接近专用说话人验证系统，同时保持自然语言接口。
原文摘要 · Abstract (English)
Speech-aware large language models (LLMs) can accept speech inputs, yet their training objectives largely emphasize linguistic content or specific fields such as emotions or the speaker's gender, leaving it unclear whether they encode speaker identity. First, we propose a model-agnostic scoring protocol that produces continuous verification scores for both API-only and open-weight models, using confidence scores or log-likelihood ratios from the Yes/No token probabilities. Using this protocol, we benchmark recent speech-aware LLMs and observe weak speaker discrimination (EERs above 20% on VoxCeleb1). Second, we introduce a lightweight augmentation that equips an LLM with ASV capability by injecting frozen ECAPA-TDNN speaker embeddings through a learned projection and training only LoRA adapters. On TinyLLaMA-1.1B, the resulting ECAPA-LLM achieves 1.03% EER on VoxCeleb1-E, approaching a dedicated speaker verification system while preserving a natural-language interface.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。