arXiv:2512.22467cs.LG2025-12

无需反向传播,用简单方法融合多个专家模型提升新领域表现

GLUE: Gradient-free Learning to Unify Experts

  • 用凸组合融合固定专家模型,通过无梯度优化学习混合系数
  • 在三个数据集上比按数据量加权提升8.5%准确率,比代理指标提升9.1%
  • 仅需两次前向传播,适合资源受限场景,可替代昂贵的反向传播训练

在多语言语音识别、跨医院影像分析、区域特定感知等部署系统中,多个预训练专家模型并存。当需要扩展至新目标域时,常需一个能超越单个专家性能的泛化模型。现有方法通过混合专家模型获得目标模型的初始参数,但启发式混合(如按数据量或代理指标加权)常导致目标域测试精度下降,而基于目标域损失函数学习混合系数通常需代价高昂的全网络反向传播。本文提出GLUE(Gradient-free Learning to Unify Experts),将目标模型初始化为固定专家的凸组合,并使用无梯度两样本SPSA更新混合系数,每步仅需两次前向传播。在三个数据集和三种网络架构上的实验表明,GLUE生成的参数先验经微调后优于基线。其测试准确率相比数据量加权提升最高达8.5%,相比代理指标选择提升最高达9.1%;在性能上,要么优于基于反向传播的全梯度混合,要么在1.4%内持平。

原文摘要 · Abstract (English)

In many deployed systems (multilingual ASR, cross-hospital imaging, region-specific perception), multiple pretrained specialist models coexist. Yet, new target domains often require domain expansion: a generalized model that performs well beyond any single specialist's domain. Given a new target domain, existing methods obtain a single strong initialization prior for the model parameters by blending expert models to initialize a target model. However, heuristic blending -- using mixing coefficients based on data size or proxy metrics -- often yields lower target-domain test accuracy, and learning these coefficients on the target domain's loss function typically requires computationally-expensive full backpropagation through a neural network. We propose GLUE, Gradient-free Learning to Unify Experts, which initializes the target model as a convex combination of fixed experts and learns the mixture coefficients of this combination via gradient-free two-point SPSA (simultaneous perturbation stochastic approximation) updates, requiring only two forward passes per step. Across experiments on three datasets and three network architectures, GLUE produces model parameter priors that can be fine-tuned to outperform baselines. GLUE improves test accuracy by up to 8.5% over data-size weighting and by up to 9.1% over proxy-metric selection. GLUE either outperforms backpropagation-based full-gradient mixing or matches its performance within 1.4%.

模型融合无梯度优化专家集成高效初始化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。