构建首个语音角色扮演评估数据集与基准，推动语音角色代理研究
SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agents
- 提出SpeechRole框架，整合大规模语音对话数据与多维度评估体系
- 覆盖98个角色、11.1万组语音对话，支持情感与语调多样性评估
- 适合语音生成、角色扮演、人机交互方向研究者使用
语音是实现逼真角色扮演的关键，但现有角色扮演代理研究主要集中于文本，导致语音角色扮演代理（SRPAs）缺乏系统性评估。我们提出SpeechRole，一个统一的SRPAs开发与评估框架。SpeechRole-Data包含98个角色和11.1万组语音到语音的对话，具有丰富的音色与韵律变化，为训练SRPAs提供大规模资源。SpeechRole-Eval提供多维度评估基准，直接衡量生成语音的副语言特征、交互能力、语音表现力和角色一致性。实验表明，端到端模型如GPT-4o Audio在流畅性与自然度上表现良好，但在韵律一致性和情绪适切性上仍受限；当前开源端到端模型在多个评估维度存在显著差距。级联与端到端系统在交互能力和角色一致性上表现相当，表明这些方面仍主要受底层文本语言模型影响。所有数据、代码与评估工具已公开于https://github.com/yuhui1038/SpeechRole。
原文摘要 · Abstract (English)
Speech is essential for realistic role-playing, yet existing work on role-playing agents largely centers on text, leaving Speech Role-Playing Agents (SRPAs) underexplored and without systematic evaluation. We introduce SpeechRole, a unified framework for developing and assessing SRPAs. SpeechRole-Data contains 98 roles and 111k speech-to-speech conversations with rich timbre and prosodic variation, providing large-scale resources for training SRPAs. SpeechRole-Eval offers a multidimensional benchmark that directly evaluates generated speech, preserving paralinguistic cues and measuring interaction ability, speech expressiveness, and role-playing fidelity. Experiments show that end-to-end SRPAs such as GPT-4o Audio achieve strong fluency and naturalness, but remain limited in prosody consistency and emotion appropriateness. In contrast, current open-source end-to-end models exhibit substantial performance gaps across multiple evaluation dimensions. Cascaded and end-to-end systems achieve comparable results in interaction ability and role-playing fidelity, suggesting that these aspects are still largely influenced by the underlying text-based language models. We release all data, code, and evaluation tools at https://github.com/yuhui1038/SpeechRole.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。