通过梯度对齐自动调节预训练损失权重，大幅降低调参成本。
When Losses Align: Gradient-Based Composite Loss Weighting for Efficient Pretraining
- 基于梯度的双层优化方法，在线学习预训练损失权重。
- 相比随机搜索，调参开销仅增加约30%的训练时间。
- 适用于事件序列建模与自监督视觉任务，效果优于人工调参。
现代深度模型常在大规模无标签数据上使用复合目标进行预训练，其中多个损失项的相对权重作为超参数。传统方法如随机搜索或贝叶斯优化需多次独立训练，计算成本高。为此，本文提出一种基于梯度的双层方法，通过将复合预训练梯度与下游目标对齐，实现预训练损失权重的在线学习。利用损失结构特性，该方法避免了截断反向传播所需的多次反向传播，将超参数调优开销控制在单次训练的约30%以内。在事件序列建模和自监督计算机视觉任务上，该方法达到或超越精心调参的基线性能，同时显著降低调参成本。
原文摘要 · Abstract (English)
Modern deep models are often pretrained on large-scale data with missing labels using composite objectives, where the relative weights of multiple loss terms act as hyperparameters. Tuning these weights with random search or Bayesian optimization is computationally expensive, as it requires many independent training runs. To address this, we propose a gradient-based bilevel method that learns pretraining loss weights online by aligning the composite pretraining gradient with a downstream objective. By exploiting the structure of the loss, the method avoids the multiple backward passes typically required by truncated backpropagation through the full model, reducing the overhead of hyperparameter tuning to approximately 30% above a single training run. We evaluate the approach on event-sequence modeling and self-supervised computer vision, where it matches or improves upon carefully tuned baselines while substantially reducing the cost of hyperparameter tuning compared to random or Bayesian search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。