提出量化模型重训成本的指标RESQUE，助力可持续模型复用。
RESQUE: Quantifying Estimator to Task and Distribution Shift for Sustainable Model Reusability
- 通过预测分布变化或任务变更时的重训成本，给出单一评估指标。
- 与重训所需周期、梯度范数、参数变化等多维度指标高度相关。
- 适合关注模型可持续性与环境影响的研究者和工程师使用。
为实现深度学习的可持续性，通过微调现有模型而非从头训练，是关键策略。本文提出表示分布偏移量化估计器（RESQUE），一种用于预测模型在任务或分布变化下重训成本的预测性量化指标。该指标提供一个简洁的综合指数，衡量重训所需资源。大量实验表明，RESQUE与多个重训度量（如训练轮次、梯度范数、参数幅度变化、能耗及碳排放）具有强相关性。该指标在新任务、多种噪声类型及不同噪声强度下均表现一致。因此，RESQUE可帮助用户决策在不同任务或分布偏移下的最优重训方案，实现更低成本、更低环境影响的模型复用。代码已开源：https://github.com/JEKimLab/AAAI2025RESQUE。
原文摘要 · Abstract (English)
As a strategy for sustainability of deep learning, reusing an existing model by retraining it rather than training a new model from scratch is critical. In this paper, we propose REpresentation Shift QUantifying Estimator (RESQUE), a predictive quantifier to estimate the retraining cost of a model to distributional shifts or change of tasks. It provides a single concise index for an estimate of resources required for retraining the model. Through extensive experiments, we show that RESQUE has a strong correlation with various retraining measures. Our results validate that RESQUE is an effective indicator in terms of epochs, gradient norms, changes of parameter magnitude, energy, and carbon emissions. These measures align well with RESQUE for new tasks, multiple noise types, and varying noise intensities. As a result, RESQUE enables users to make informed decisions for retraining to different tasks/distribution shifts and determine the most cost-effective and sustainable option, allowing for the reuse of a model with a much smaller footprint in the environment. The code for this work is available here: https://github.com/JEKimLab/AAAI2025RESQUE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。