提出用户中心框架Ouvia，评估语音翻译在真实交流中的可用性。
Ouvia: A User-centered Framework for Measuring Usability of Speech Translation in Real-World Communication Scenarios
- 基于真实场景的多阶段实验，收集1750+次跨语言交流数据
- 仅约一半翻译交互被评作可用，不同人群间差异显著
- 基于问答的质量评估比传统方法更能预测实际可用性
语音翻译（ST）在用户应用中日益普及，但现有评估多依赖脱离语境的测试集和整体质量指标，忽视用户真实沟通需求。本文提出Ouvia，一种面向真实世界情境下用户感知可用性的评估框架。聚焦一对一交流：英语使用者需向葡萄牙语使用者传达请求，由系统自动翻译。通过定制网页应用与多阶段研究设计，我们在医疗与日常场景中收集了超过1,750次交互数据，涉及四种ST系统、三种英语方言及两种性别。结果发现，现代语音翻译仅有限支持用户交流——约一半交互被评为可用，且不同人群间存在显著差异。此外，在各类质量指标中，基于问答的评估对实际可用性的预测能力远超传统方法。研究强调，应采用情境化、以用户为中心的评估框架，超越整体质量评分，关注技术服务于谁以及服务效果如何。
原文摘要 · Abstract (English)
Speech translation (ST) is increasingly adopted in user applications, yet its evaluation largely focuses on decontextualized testbeds and holistic quality, rather than end users' communication needs. We introduce Ouvia, an evaluation framework for measuring user-perceived usability of speech translation outputs in real-world settings. Ouvia focuses on one-to-one communication: an English speaker needs to convey a request to a Portuguese speaker, and the message is automatically translated. Through a custom web app and multi-phase study design, we collect more than 1,750 such interactions in healthcare and everyday situations, mediated by four ST systems, involving speakers from three English dialects and two genders. We find that modern ST serves people only to a limited extent -- only around half of interactions are rated as usable -- with significant gaps in reported usability across demographic groups. Moreover, among quality metrics, we find that QA-based evaluation is a substantially stronger predictor of real-world usability than standard approaches. Together, these findings stress the importance of situated, user-centered evaluation frameworks that go beyond holistic quality scores and attend to who the technology serves -- and how well.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。