提出长期可持续性评估方法,更真实反映模型更新的环境成本。
A robust methodology for long-term sustainability evaluation of Machine Learning models
- 基于在线学习思想,通过持续增量训练评估长期可持续性
- 实验证明传统静态评估会误判模型维护的真实碳成本
- 适合关注AI环境影响的研究者与企业决策者
可持续性与效率已成为人工智能系统研发与部署中的关键考量,但现有的绿色AI监管实践仍缺乏标准化、模型无关的评估协议。当前机器学习可持续性审计流程及研究惯例存在三大缺陷:1)过度强调训练轮次/批次设置;2)未正式建模模型持续适应与重训练的长期成本;3)仅衡量封闭实验的可持续性,而非真实世界中长期运行的环境影响。本文提出一种新型评估协议,借鉴在线学习理念,通过并行真实数据采集与持续增量重训练,衡量模型的可持续性与性能。在多种任务与模型类型上进行实验表明,传统静态训练-测试评估无法可靠捕捉动态数据下的可持续性,常高估、低估或不规律估计模型维护的实际成本。该评估流程还初步揭示,在真实长期运行中,更高的环境成本有时几乎不带来性能提升。
原文摘要 · Abstract (English)
Sustainability and efficiency have become essential considerations in the development and deployment of Artificial Intelligence systems, but existing regulatory practices for Green AI still lack standardized, model-agnostic evaluation protocols. Recently, sustainability auditing pipelines for ML and usual practices by researchers show three main pitfalls: 1) they disproportionally emphasize epoch/batch learning settings, 2) they do not formally model the long-term sustainability cost of adapting and re-training models, and 3) they effectively measure the sustainability of sterile experiments, instead of estimating the environmental impact of real-world, long-term AI lifecycles. In this work, we propose a novel evaluation protocol for assessing the long-term sustainability of ML models, based on concepts inspired by Online ML, which measures sustainability and performance through incremental/continual model retraining parallel to real-world data acquisition. Through experimentation on diverse ML tasks using a range of model types, we demonstrate that traditional static train-test evaluations do not reliably capture sustainability under evolving datasets, as they overestimate, underestimate and/or erratically estimate the actual cost of maintaining and updating ML models. Our proposed sustainability evaluation pipeline also draws initial evidence that, in real-world, long-term ML life-cycles, higher environmental costs occasionally yield little to no performance benefits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。