arXiv:2601.06649cs.LGcs.AI2026-01被引 1

增加训练数据量不总能提升效率,反而更耗能。

Revisiting Training Scale: An Empirical Study of Token Count, Power Consumption, and Parameter Efficiency

  • 固定硬件下测试不同数据量训练效果,关注能耗与时间
  • 数据量翻倍时效率下降,性能提升却越来越慢
  • 适合关注模型训练成本与可持续性的研究者

机器学习研究质疑了训练数据量增加是否能带来大语言模型性能的线性提升。本研究基于先前提出的能源感知参数效率指标,通过在固定硬件和训练条件下进行实证分析,考察训练数据量增加的影响。研究创新性地将功耗与执行时长(通过功耗采样频率体现)纳入数据规模分析,弥补了以往研究仅关注性能而忽略计算与能源成本的不足。在相同GPU实例、模型架构、优化器设置和训练轮次下,对11亿参数的TinyLlama模型分别在50万、100万和200万条训练样本下进行训练。尽管传统性能指标在不同数据量下表现不一或呈现边际递减,但结合能耗与时长后,发现训练效率随数据量增加呈严格单调下降趋势。重复测量方差分析显示,数据量对参数效率有显著影响,且所有两两比较在邦弗朗尼校正后仍显著。结果表明,即使边际性能略有提升,增加训练数据量也可能在能源上变得低效,强调了在大模型训练中采用效率导向评估的重要性。

原文摘要 · Abstract (English)

Research in machine learning has questioned whether increases in training token counts reliably produce proportional performance gains in large language models. Building on prior work introducing an energy-aware parameter efficiency metric, this study empirically examines the effects of increasing training token counts under fixed hardware and training conditions. The significance of this work lies in the explicit integration of power consumption and execution duration, as reflected by the power sampling frequency, into token-scale analysis. This addresses a gap in prior studies emphasizing performance outcomes while underrepresenting computational and energy costs. Using a repeated-measures experimental design on a constant GPU instance with an identical model architecture, optimizer settings, and epoch counts, a 1.1-billion-parameter TinyLlama model was trained at three token counts (500K, 1M, and 2M). While conventional performance metrics exhibited inconsistent or diminishing returns across token scales, the inclusion of power consumption and execution duration revealed a strictly monotonic decline in training efficiency as token count increased. Repeated-measures ANOVA demonstrated a strong effect of token count on parameter efficiency, with all pairwise comparisons remaining significant following Bonferroni correction. These findings indicate that increases in training token counts may be energetically inefficient even when marginal performance improvements are observed, underscoring the importance of efficiency-aware evaluation in large language model training.

训练效率能耗分析大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。