arXiv:2605.30792eess.AScs.AI2026-05被引 2

统一评估语音翻译的多维度表现,打破质量指标孤立评价的局限。

OpenSTBench: Beyond Semantic Evaluation for Speech Translation

  • 构建统一框架,整合文本、语音与时间维度的评估标准。
  • 实验证明高翻译质量系统在语音与时间表现上仍有显著差异。
  • 适合语音翻译研发者与应用选型者进行系统对比分析。

语音翻译系统涵盖语音转文本(S2TT)、语音转语音(S2ST)、离线与流式生成等多种形式,输出在模态、语音实现和时序行为上差异显著。现有评估方法分别考察翻译质量、语音质量和时序质量,但各自独立,难以全面比较异构系统。为此,我们提出OpenSTBench,一个统一的多维评估框架,将异构语音翻译输出纳入统一格式。该框架支持离线与流式场景下的S2TT与S2ST系统,联合评估翻译质量、语音质量、说话人保留度、情感与副语言保真度、时序一致性及延迟。在代表性系统上的实验表明,翻译质量强的系统在语音质量与时序表现上仍存在显著差异。OpenSTBench提供了可复现的协议,用于分析这些跨维度差异,并支持面向应用的系统比较。代码与数据集见https://github.com/sjtuayj/OpenSTBench。

原文摘要 · Abstract (English)

Speech translation systems increasingly span speech-to-text translation (S2TT), speech-to-speech translation (S2ST), offline translation, and streaming generation, producing outputs that differ in modality, speech realization, and timing behavior. Existing evaluation practices assess important aspects such as translation quality, speech quality, and temporal quality, but these aspects are often evaluated under separate protocols, making it difficult to compare heterogeneous systems comprehensively. To address this gap, we present OpenSTBench, a unified multidimensional evaluation framework that organizes heterogeneous speech translation outputs into a shared evaluation format. OpenSTBench supports both S2TT and S2ST systems in offline and streaming settings, and jointly evaluates translation quality, speech quality, speaker preservation, emotion and paralinguistic fidelity, temporal consistency, and latency. Through experiments on representative speech translation systems, we show that systems with strong translation quality can still differ substantially in speech quality, as well as in temporal quality. OpenSTBench provides a reproducible protocol for analyzing these cross-dimensional differences and supporting application-oriented comparison of speech translation systems. The code and datasets are available at https://github.com/sjtuayj/OpenSTBench.

语音翻译多维评估开放基准系统对比

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。