首个评估语音指令跟随中语气表达的基准，让语音模型更像真人交流。
S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Models
- 构建四层复杂度递增的语音交互协议，测试模型对语气等非语言信息的理解
- 生成1243段真实任务语音样本，覆盖100多个场景，支持直接语音对比评测
- 发现当前主流语音模型在复杂语气任务上表现差距大，揭示提升表达力的关键设计
大型语言模型的进展重塑了语音到语音（S2S）系统，使其日益接近自然对话。然而，现有评测仍以文本为主，忽略韵律、情感和说话人特征等关键非语言线索，而这些正是富有表现力和类人沟通的核心。我们提出 S2S-Arena，一个以语音为本的评测基准，专门评估指令跟随型 S2S 模型在语义理解与非语言表达两方面的表现。该基准包含四级逐步增强非语言复杂度的交互协议，采用两阶段数据构建流程，生成涵盖100多个真实任务的1,243段语音样本，并设计了无需参考、直接在语音模态进行成对比较的竞技场式评估框架。对10个顶尖S2S系统的超1000次比较显示，当前学术与工业系统在复杂非语言需求下存在显著性能差距。分析进一步揭示了影响表达性指令遵循的关键设计因素，为打造更自然、鲁棒且与人类对齐的语音代理提供可操作洞见。
原文摘要 · Abstract (English)
Recent advances in large language models (LLMs) have fundamentally reshaped speech-to-speech (S2S) systems, enabling increasingly natural spoken interaction. However, existing benchmarks still rely heavily on text-based evaluation and largely ignore paralinguistic cues such as prosody, emotion, and speaker traits, which are central to expressive and human-like communication. We introduce S2S-Arena, a speech-native benchmark for evaluating instruction-following S2S models with explicit assessment of both semantic understanding and paralinguistic expression. S2S-Arena features a four-level interaction protocol that systematically probes models under increasing paralinguistic complexity, a two-stage data construction pipeline that produces 1,243 speech samples spanning 100+ real-world tasks, and an arena-style evaluation framework that enables reference-free, pairwise comparison directly in the speech modality. Benchmarking 10 state-of-the-art S2S systems over 1,000+ comparisons reveals substantial performance gaps (especially under complex paralinguistic demands) between current academic and industrial systems. Our analysis further identifies key design factors governing expressive instruction following, providing actionable insights for building more natural, robust, and human-aligned speech agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。