通过相关性排列对齐语音音乐编码器，实现高效融合。
A correlation-permutation approach for speech-music encoders model merging
- 基于特征相关性计算层间排列矩阵，对齐不同模型内部结构。
- 融合后音乐性能提升14.83分，保留语音识别能力。
- 适用于独立训练的语音与音乐编码器快速合并。
构建统一的语音与音乐模型通常需要昂贵的预训练。模型融合可在极低计算成本下实现统一音频模型。然而，当模型权重空间未对齐时，直接融合面临挑战。受Git Re-Basin启发，我们提出一种相关性-排列方法，将音乐编码器的内部层与语音编码器对齐。该方法扩展至Transformer层的融合场景，通过逐层计算最大化特征间交叉相关性的排列矩阵，实现原本不兼容模型的有效融合。合并后的模型在保留语音能力的同时显著提升音乐性能，在平均得分上相比线性插值融合方法提高14.83分。本工作使从独立训练的编码器创建统一音频模型成为可能。
原文摘要 · Abstract (English)
Creating a unified speech and music model requires expensive pre-training. Model merging can instead create an unified audio model with minimal computational expense. However, direct merging is challenging when the models are not aligned in the weight space. Motivated by Git Re-Basin, we introduce a correlation-permutation approach that aligns a music encoder's internal layers with a speech encoder. We extend previous work to the case of merging transformer layers. The method computes a permutation matrix that maximizes the model's features-wise cross-correlations layer by layer, enabling effective fusion of these otherwise disjoint models. The merged model retains speech capabilities through this method while significantly enhancing music performance, achieving an improvement of 14.83 points in average score compared to linear interpolation model merging. This work allows the creation of unified audio models from independently trained encoders.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。