arXiv:2605.25499cs.LG2026-05

提出高效动态重要性加权框架,提升模型在分布偏移下的训练效率与性能。

Accelerated Dynamic Importance Weighting with Versatile Divergence-Minimizing Estimators

论文配图:Accelerated Dynamic Importance Weighting with Versatile Divergence-Minimizing Estimators
图 1 · 摘自论文原文
  • 采用轻量级梯度更新,基于历史权重快速优化,减少每轮计算开销。
  • 在多个数据集上实现最优加权效果,较原方法提速3倍以上。
  • 支持多种散度度量的插件式加权方法,适用于不同场景的分布偏移问题。

重要性加权(IW)是解决联合分布偏移问题的有效方法,即训练与测试数据的联合分布不一致。传统方法通过估计测试到训练的密度比作为权重,重新加权训练损失。近年来,动态重要性加权(DIW)将权重估计融入模型训练,实现了深度模型的可扩展性,在现代大规模数据集上表现优异。然而,现有DIW仍存在两大局限:一是每小批量需收敛求解核均值匹配(KMM)优化问题,带来显著计算开销;二是仅依赖KMM进行权重估计,而重要性加权文献中存在基于不同散度度量的多样化方法。本文提出加速动态重要性加权(ADIW),一种统一且高效的深度学习分布偏移处理框架。ADIW通过少量轻量级投影梯度更新,以先前权重为起点进行暖启动,大幅提升效率。同时,将DIW推广为统一的散度最小化框架,支持包括KL散度、平方距离和Wasserstein-1距离在内的多种加权估计方法,实现即插即用。我们在温和条件下建立了ADIW的收敛性保证,实验证明其在保持最佳加权性能的同时,显著提升计算效率,达到当前最优水平。

原文摘要 · Abstract (English)

Importance weighting (IW) is a golden solver for joint distribution shift, where the joint distributions differ between the training and test data. To solve this problem, IW estimates test-to-training density ratios as importance weights and reweights the training losses accordingly. Recent advances in dynamic IW (DIW) integrate weight estimation into model training, enabling scalable IW for deep models and achieving strong performance on large modern datasets. Despite its promise, DIW remains limited in two aspects. First, it incurs substantial computational overhead by solving a kernel mean matching (KMM)-induced optimization problem to convergence in every mini-batch. Second, it relies solely on KMM for weight estimation, whereas the IW literature contains diverse estimation methods based on different divergence measures. In this paper, we propose accelerated DIW (ADIW), a unified and efficient IW framework for deep learning under joint distribution shift. ADIW performs a few lightweight projected gradient descent updates that warm-start from previously updated weights, substantially improving efficiency. Moreover, ADIW generalizes DIW into a unified divergence-minimization framework that supports diverse weight-estimation methods in a plug-and-play manner, including those based on the Kullback-Leibler divergence, squared distance, and Wasserstein-1 distance. We establish convergence guarantees for ADIW under mild conditions, and empirical results demonstrate that ADIW achieves state-of-the-art IW performance while being substantially more efficient.

重要性加权分布偏移高效训练散度最小化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。