给微调加碳排放惩罚,让大模型推理更省电且不丢性能。
Can We Optimize the Performance-Carbon Emission Break-Even Point?: The Quest for Greener LLMs

- 在微调时加入可微的碳排放代理模型,实时控制能耗。
- 在三个任务上实现性能提升的同时碳排放基本不变或降低。
- 适合关注绿色AI、想低成本优化模型能耗的研究者。
任何部署的大语言模型(LLM)在推理阶段产生的碳足迹会随使用次数累积,远超微调的一次性成本。然而,现有效率优化主要针对预训练规模或后期压缩。本文探讨能否将校准过的、可微的能源代理模型融入微调目标,使模型在推理时以零或近零碳成本获得任务准确率提升,即达到性能-碳排放的平衡点。我们提出一种联合损失机制,包含每个模型的碳排放参数,基于参数量、计算量(FLOP)和内存占用的线性代理模型,并通过实际硬件能效测试拟合。在三种不同架构的模型(Gemma-2 2B、Llama-3.1 8B、Qwen-2.5 14B)上进行微调,并在三个MMLU子任务(抽象代数、哲学、形式逻辑)上评估推理F1得分与二氧化碳排放。结果表明,碳项在不同任务中可能表现为有害干扰或有益正则化。研究发现,校准后的碳感知微调可作为轻量级、即插即用的正则化手段,在模型和任务依赖的范围内存在非空的平衡区域。本工作仍在持续中,代码库即将发布。
原文摘要 · Abstract (English)
The carbon footprint of any deployed Large Language Model (LLM) accumulates during inference, where repeated use of the model substantially exceeds the one-time cost of fine-tuning. Yet most efficiency interventions target either pre-training scale or post-hoc compression. We ask whether folding a calibrated, differentiable energy surrogate into the fine-tuning objective can produce inference behavior that gains task accuracy at zero or near-zero carbon cost, a break-even configuration. We propose a joint loss mechanism with a per-model carbon-emission parameter, a linear surrogate over parameter norm, FLOP proxy, and a memory proxy, fit from on-hardware energy profiling. We fine-tune three architecturally distinct families: Gemma-2 2B, Llama-3.1 8B, and Qwen-2.5 14B, and evaluate inference F1 and CO$_2$ emissions on three MMLU subjects: abstract algebra, philosophy, and formal logic. We discover from several outcomes that the carbon term behaves as either harmful interference or beneficial regularization depending on the task structure. We position calibrated carbon-aware fine-tuning as a lightweight, drop-in regularizer with a non-empty but model and task-dependent break-even region. This is an ongoing work, and we will release our codebase soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。