用经济学思维优化大模型计算效率，让模型更省力、更透明。
Computational Economics in Large Language Models: Exploring Model Behavior and Incentive Design under Resource Constraints
- 把注意力头和神经元块当作资源有限的经济个体，动态分配计算资源。
- 在GLUE和WikiText-103上实现40%的FLOPS降低，精度不降且推理更快。
- 适合关注模型效率、可解释性与资源受限部署的研究者与工程师。
大语言模型（LLMs）面临高昂的计算成本。本文提出“计算经济学”框架，将LLM视为由资源受限的代理（注意力头与神经元块）构成的内部经济系统，需在有限计算下最大化任务效用。实证发现，当计算稀缺时，标准LLM会将注意力集中于高价值标记，同时保持准确率。基于此，我们提出一种激励驱动训练范式，在任务损失中加入可微的计算成本项，促使激活稀疏化与高效化。在GLUE（MNLI、STS-B、CoLA）和WikiText-103数据集上，该方法生成的模型群呈现帕累托前沿，始终优于后处理剪枝；在相近精度下，实现约40%的FLOPS减少与更低延迟，且注意力模式更具可解释性。结果表明，经济原则为资源严苛约束下的高效、自适应与透明模型设计提供了合理路径。
原文摘要 · Abstract (English)
Large language models (LLMs) are limited by substantial computational cost. We introduce a "computational economics" framework that treats an LLM as an internal economy of resource-constrained agents (attention heads and neuron blocks) that must allocate scarce computation to maximize task utility. First, we show empirically that when computation is scarce, standard LLMs reallocate attention toward high-value tokens while preserving accuracy. Building on this observation, we propose an incentive-driven training paradigm that augments the task loss with a differentiable computation cost term, encouraging sparse and efficient activations. On GLUE (MNLI, STS-B, CoLA) and WikiText-103, the method yields a family of models that trace a Pareto frontier and consistently dominate post-hoc pruning; for a similar accuracy we obtain roughly a forty percent reduction in FLOPS and lower latency, together with more interpretable attention patterns. These results indicate that economic principles offer a principled route to designing efficient, adaptive, and more transparent LLMs under strict resource constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。