评测语音识别对关键数值的精确还原能力,发现传统指标有盲区。
VoiceCodeBench: Evaluating Exact Structured-Token Recovery in Automatic Speech Recognition
- 构建包含1482个目标实体的语音基准测试集,聚焦精确值恢复。
- 12个基线系统最高任务成功率仅68.7%,近三分之一关键值丢失。
- 建议用结构化实体匹配等新指标替代纯字错误率评估生产级系统。
自动语音识别(ASR)系统通常以词错误率(WER)评估,但许多语音工作流依赖于标识符、路径和测量值的精确文本。一段转录可能流畅且拥有低WER,却破坏了下游系统必须解析、存储或执行的关键值。我们提出VoiceCodeBench,一个用于评估英文ASR中精确结构化标记恢复的基准。该数据集包含300段人工录制的工作场景音频,覆盖八个业务领域,共1482个经审核的目标实体,分属26种类型,每项均有可从音频中恢复的标准书写形式。在仅提供原始音频的协议下,系统接收音频字节,无额外上下文或元数据。除WER外,还评估标准标记/实体匹配率(CTEM)、任务成功率达(TSR)及按类型精确恢复率。12个基线系统中,较低的WER普遍对应更好的结构化标记恢复,但未完全决定结果:WER与CTEM、TSR的斯皮尔曼相关系数分别为-0.73。最强基线的TSR仅为68.7%,近三分之一录音存在至少一个未恢复的关键值。结果表明,需采用实体敏感指标来评估ASR输出是否保留了生产系统必须解析、路由、存储、比较或执行的精确值。
原文摘要 · Abstract (English)
Automatic speech recognition (ASR) systems are commonly evaluated with word error rate (WER), yet many voice workflows depend on exact written values for identifiers, paths, and measured quantities. A transcript can appear fluent and achieve low WER while corrupting a value that a downstream system must parse, store, or execute. We introduce VoiceCodeBench, a benchmark for evaluating exact structured-token recovery in English ASR. It contains 300 human-recorded workplace segments spanning eight workflow domains and 1,482 audited target entities across 26 entity types, each with a canonical written form recoverable from the audio. Under a raw-audio-only protocol, systems receive audio bytes without additional context or metadata. Alongside WER, we evaluate Canonical Token/Entity Match (CTEM), Task Success Rate (TSR), and per-type exact recovery. Across 12 baseline ASR systems, lower WER generally corresponded to better structured-token recovery but did not fully determine it: Spearman correlations were -0.73 for both WER versus CTEM and WER versus TSR. The strongest baseline by TSR reached only 68.7%, leaving nearly one third of recordings with at least one unrecovered workflow-critical value. These results show that entity-sensitive metrics are needed to assess whether ASR output preserves exact values that production systems must parse, route, store, compare, or execute.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。