构建音频编码器评估基准，统一语义与声学标记定义
AudioCodecBench: A Comprehensive Benchmark for Audio Codec Evaluation
- 提出语义与声学标记的清晰定义，解决现有研究混淆问题
- 从重建质量、码本索引稳定性等四维度系统评估编码器性能
- 适用于大模型语音音乐处理，帮助选型与性能对比
多模态大语言模型在语音与音乐领域广泛应用，推动了面向大模型的音频分词研究。与仅关注语义的文本标记不同，音频标记需同时捕捉全局语义和细微声学细节，并为语音与音乐提供可有效融入多模态大模型的离散表示。然而，现有研究对语义标记与声学标记的定义不明确。此外，现有编码器评估多集中于特定任务（如重建或自动语音识别），难以进行公平全面比较。为此，本文提出合理的语义与声学标记定义，并构建系统性评估框架，涵盖四个维度：音频重建指标、码本索引（ID）稳定性、仅解码器变换器困惑度，以及下游探针任务表现。实验验证了定义的合理性及各指标间的相关性。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have been widely applied in speech and music. This tendency has led to a focus on audio tokenization for Large Models (LMs). Unlike semantic-only text tokens, audio tokens must both capture global semantic content and preserve fine-grained acoustic details. Moreover, they provide a discrete method for speech and music that can be effectively integrated into MLLMs. However, existing research is unsuitable in the definitions of semantic tokens and acoustic tokens. In addition, the evaluation of different codecs typically concentrates on specific domains or tasks, such as reconstruction or Automatic Speech Recognition (ASR) task, which prevents fair and comprehensive comparisons. To address these problems, this paper provides suitable definitions for semantic and acoustic tokens and introduces a systematic evaluation framework. This framework allows for a comprehensive assessment of codecs' capabilities which evaluate across four dimensions: audio reconstruction metric, codebook index (ID) stability, decoder-only transformer perplexity, and performance on downstream probe tasks. Our results show the correctness of the provided suitable definitions and the correlation among reconstruction metrics, codebook ID stability, downstream probe tasks and perplexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。