基于曲率加权的模型层容量分配，让大模型优化更精准高效。
Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer-Adaptive Large Language Model Optimization
- 用曲率信息构建层重要性评分，替代传统梯度方法。
- 在硬件预算下实现最优专家分配与剪枝比例，提升性能。
- 理论保证稳定迁移，适合模型压缩与资源受限场景。
大语言模型各层容量分布极不均衡：部分层对损失下降贡献显著，而其他层几乎冗余。现有层评分方法仅提供敏感性估计，无法在全局硬件预算下给出合理的分配或剪枝决策依据。本文提出一种基于曲率感知、受最小描述长度启发的框架，核心为层增益量 $ζ_k^2 = g_k^ op\widetilde H_{kk}^{-1}g_k$，该量等于正则化层受限二次模型预测的最大下降值，包含局部逆曲率信息，是可减少风险的局部代理，而非对梯度范数得分的普遍超越。将增益归一化为评分 $q_k$ 后，构建两个凸规划：一个在递减收益下分配专家槽位，另一个在保护高分层前提下设定层级剪枝比例。两者均具唯一全局最优解，可通过二分法在 $O(K\log(1/\varepsilon))$ 时间内求解。还证明了二次转移遗憾界：当源与目标评分向量差异不超过 $δ$ 时，转移决策的目标代理代价与最优值相差 $O(δ^2)$。在 Mistral-7B 与 Gemma-7B 上的实验显示，某些设置下分配性能明显提升，剪枝表现竞争但结果混合。因此，该框架将经验性的评分到决策的启发式方法，替换为具有预算可行性和理论保证的优化过程，适用于所述连续代理。
原文摘要 · Abstract (English)
Layer-wise capacity in large language models is highly non-uniform: some layers contribute disproportionately to loss reduction, whereas others are nearly redundant. Existing layer-scoring methods provide sensitivity estimates but do not give a principled rule for converting those estimates into allocation or pruning decisions under a global hardware budget. We introduce a curvature-aware, MDL-inspired framework built around the layer gain $ζ_k^2=g_k^\top\widetilde H_{kk}^{-1}g_k$. This quantity equals twice the maximal decrease predicted by the regularized layer-restricted quadratic model and incorporates inverse local curvature; it is therefore a local surrogate for reducible risk, not a universal dominance claim over gradient-norm scores. After normalizing the gains into scores $q_k$, we formulate two convex programs: one allocates expert slots under diminishing returns, and the other assigns layer-wise pruning ratios while protecting high-score layers. Both continuous programs have unique globally optimal solutions characterized by one dual variable and computable in $O(K\log(1/\varepsilon))$ time by bisection. We also prove a quadratic transfer-regret bound: when source and target score vectors differ by at most $δ$, the target surrogate cost of the transferred decision is within $O(δ^2)$ of the target optimum. Experiments on Mistral-7B and Gemma-7B show clear allocation gains in some settings and competitive, though mixed, pruning performance. The framework therefore replaces an empirical score-to-decision heuristic with a budget-feasible optimization procedure whose guarantees apply to the stated continuous surrogates. Code is available on github repo - [TKAI-LAB-Mali/Curvature-Weighted-Capacity-Allocation](https://github.com/TKAI-LAB-Mali/Curvature-Weighted-Capacity-Allocation.git)
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。