用音频验证修复语音对话状态追踪中的三类错误
AVERT: Audio-Verified Adjudication for Spoken Dialogue State Tracking
- 结合跨轮次一致性和音频验证器，动态修正语音识别错误
- 在SpokenWOZ上达40.13的JGA，接近10亿参数端到端模型表现
- 仅需两个10亿参数解码器，无需重新训练即可显著提升准确率
语音对话状态追踪需从语音中恢复槽位-值对，其中语音识别错误主要集中在实体值且跨轮持续存在，使其成为生成与编辑双重挑战。现有逐轮文本编辑器可纠正大量错误，但仅基于转录文本，仍存在三类可修复误差：跨轮不一致的值、遗漏的槽位以及音频不支持的值。本文提出AVERT，通过融合跨轮一致性与训练好的音频条件验证器，对候选值进行评分，并采用三种操作（投票、添加、替换）分别针对常见错误类型，且每种操作限定于特定槽位。在SpokenWOZ数据集上，基础语音大模型达33.04 JGA，文本编辑器提升至38.34，而AVERT达到40.13，无需重训练。其性能接近使用完整语音历史的10亿参数端到端系统（39.32），但仅依赖两个10亿参数解码器。音频验证器带来统计显著增益，且限制操作作用槽位至关重要：若取消限制，无差别投票会覆盖正确分类值，导致性能低于编辑器。
原文摘要 · Abstract (English)
Spoken dialogue state tracking recovers slot-value pairs from speech, where ASR errors concentrate in entity values and persist across turns, making it both a generation and an editing problem. A strong per-turn text editor corrects much of this but, operating on the transcript alone, leaves three recoverable errors: a value predicted inconsistently across turns, an omitted slot, and a value the audio does not support. We present AVERT, which scores each candidate value by combining cross-turn agreement with a trained audio-conditioned verifier and resolves the three error types with three operators, vote, add, and swap, each restricted to the slots where its error is common. On SpokenWOZ, a base speech-LLM reaches 33.04 JGA, a text editor 38.34, and AVERT 40.13, without retraining either. This is in the range of a 1B end-to-end system that consumes the full spoken history (39.32), though AVERT uses two 1B decoders rather than one. The audio verifier contributes a statistically significant gain, and restricting each operator to a selected slot subset matters: removing it lets unrestricted voting overwrite correct categorical values and fall below the editor.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。