分两步建模音频表示，提升跨领域迁移能力。
BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations
- 先上下文编码再预测掩码区域，分离建模流程
- 音乐和环境音表现显著提升，语音性能微降
- 推理计算量不变，适合部署到资源受限场景
自监督学习使音频表示具备跨领域和任务的迁移能力。我们提出 BEST-RQ-2，是 BEST-RQ 的演进版本，保留了冻结的随机投影离散目标，引入两阶段‘上下文建模-预测’预训练机制。视觉变换器（ViT）仅处理未掩码的频谱区域，轻量级预测器则推断掩码区域的目标，预训练后丢弃预测器。用 ViT 替代原 Conformer 编码器后，整体性能在多个领域发生转移：语音性能略有下降，但音乐和环境声音性能提升，平均得分相近。主要改进来自将掩码预测拆分为独立的上下文建模与预测阶段。在 X-ARES 与 XARES-LLM 基准测试中，BEST-RQ-2 持续优于单阶段基线模型，且推理计算量保持不变。代码与模型检查点已公开。
原文摘要 · Abstract (English)
Self-supervised learning enables audio representations that transfer across domains and tasks. We present BEST-RQ-2, an evolution of BEST-RQ that retains frozen randomprojection-based discrete targets while introducing a two-step contextualize-then-predict pretraining scheme. A ViT context encoder processes only the unmasked spectrogram regions, and a lightweight predictor infers targets for the masked regions; the predictor is discarded after pretraining. Replacing the original Conformer encoder with a ViT shifts performance across domains, slightly reducing speech performance while improving music and environmental sounds, with comparable average scores. The main improvement comes from decomposing masked prediction into separate contextualization and prediction stages. On the X-ARES and XARES-LLM benchmarks, BEST-RQ-2 consistently outperforms one-stage baselines in overall transfer while keeping inference compute unchanged. Code and model checkpoints are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。