arXiv:2511.01077cs.LGstat.CO2025-11NeurIPS被引 4

别只看模型大小,该看每单位资源能带来多少能力。

AI Progress Should Be Measured by Capability-Per-Resource, Not Scale Alone: A Framework for Gradient-Guided Resource Allocation in LLMs

  • 根据参数梯度影响识别关键部分,只更新高影响力参数
  • 仅更新关键参数可提升性能-资源比,节省大量算力
  • 适合关注效率、环保和资源公平的开发者与研究者

本文挑战当前人工智能研究中过度追求模型规模与计算量的“扩展教条”,指出其导致环境负担加重与资源分配不均。提出应以“单位资源能力”为核心评估标准。理论分析显示,在基于Transformer的模型中,少数参数具有显著影响(服从重尾分布),关键发现有三:(1) 仅更新高影响参数在性能-资源比上优于全参数微调;(2) 梯度范数可作为高效识别高影响组件的代理指标;(3) 参数与数据协同选择能实现乘法级效率提升,可能使资源需求降低数个数量级。据此提出双阶段范式:基础模型开发采用边际回报预训练,下游用户使用影响引导适配,并通过梯度蓝图(包含各任务关键参数的元数据)衔接。这一视角将原本的硬件权宜之计升华为理论最优策略,推动AI向更可持续、更公平的方向演进。

原文摘要 · Abstract (English)

This position paper challenges the "scaling fundamentalism" dominating AI research, where unbounded growth in model size and computation has led to unsustainable environmental impacts and widening resource inequality. We argue that LLM development should be fundamentally reoriented toward capability-per-resource rather than capability alone. We present a theoretical framework demonstrating that resource-allocation decisions guided by gradient influence patterns can dramatically improve efficiency throughout the AI lifecycle. Our analysis shows that in transformer-based models, where a small fraction of parameters exert outsized influence (following heavy-tailed distributions), three critical insights emerge: (1) updating only high-influence parameters strictly outperforms full-parameter tuning on a performance-per-resource basis; (2) simple gradient norms provide computationally efficient proxies for identifying these high-influence components; and (3) coordinated parameter and data selection yields multiplicative efficiency gains, potentially reducing resource requirements by orders of magnitude. Building on these theoretical foundations, we propose a two stage paradigm marginal-return pretraining for foundation developers and influence guided adaptation for downstream users bridged by gradient blueprints, metadata describing which parameters matter most for various tasks. This capability-per-resource perspective transforms what were once considered pragmatic hardware workarounds into theoretically optimal strategies, democratizing access to cutting-edge AI capabilities while significantly reducing environmental impact. By embedding resource consciousness into how we develop, adapt, and evaluate models, we can reshape AI progress toward a more sustainable and equitable future.

效率优化模型压缩可持续AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。