一个模型同时搞定语音识别、说话人分离和说话人辨认，无需微调。
Echo: A Joint-Embedding Predictive Architecture for Speaker Diarization and Speech Recognition in a Shared Latent Space
- 用统一的512维隐空间联合编码说话人身份、语音内容和动态源路由。
- 在未知说话人数量的合成数据上，盲辨认错误率15.00%，分离准确率97.80%。
- 适合研究多任务音频建模或轻量化系统部署的开发者参考。
我们提出Echo，一个基于单个2500万参数ViT编码器的音频系统原型。该编码器通过JEPA目标预训练，随后分阶段优化以在同一512维隐空间中同时表征说话人身份、音素内容与动态源路由，部署时无需针对任务微调。轻量级头部处理说话人辨认（ArcFace + VBx）与动态源分离（空目标K集预测）。在未知说话人数量的合成VoxCeleb2混合语音上，标准架构达到15.00%盲辨认错误率（DER）、97.80% PIT分离准确率，隐空间信噪比提升+9.52 dB，且在保留的k-NN探测中，说话人与内容因子分解差距达+53.50点。Echo的核心并非单一任务的新最优，而是以如此小的模型规模实现三项任务的共存。我们逐阶段记录设计过程，报告失败路径，并指出端到端语音识别仍受制于VQ瓶颈这一原型局限。
原文摘要 · Abstract (English)
We present Echo, a proof-of-concept audio system built around a single 25 M-parameter ViT encoder. The encoder is pretrained with a JEPA objective and then specialised by stages to carry speaker identity, phonetic content, and dynamic source routing in the same 512-dimensional latent space, with no per-task fine-tuning at deployment. Light heads handle diarization (ArcFace + VBx) and dynamic source separation (null-target K-set prediction). On synthetic VoxCeleb2 mixtures with unknown K, the canonical stack reaches 15.00% blind DER, 97.80% PIT separation accuracy with +9.52 dB latent SI-SDR, and a +53.50-point speaker/content factorisation gap on a held-out k-NN probe. The point of Echo is not a new SOTA on any single task but the joint coexistence of three tasks on one encoder at this footprint. We document the design stage by stage, report the dead-ends, and identify the structural wall on end-to-end ASR through the VQ bottleneck that still bounds the PoC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。