arXiv:2509.17874cs.LG2025-09被引 4

让大模型推理时可动态调节计算量,性能与效率自由切换。

Deep Hierarchical Learning with Nested Subspace Networks for Large Language Models

  • 通过嵌套子空间重构线性层,实现计算量连续可调。
  • 同一模型在50%减少算力下仅损失5%准确率。
  • 适合资源受限或需灵活部署的大型语言模型场景。

大型神经网络通常在固定计算预算下训练,导致性能与效率之间存在僵化权衡,不适用于资源受限或动态环境。现有方法面临两难:训练多个专用模型计算成本过高,而可剪枝网络等动态方法难以应用于大型预训练基础模型。本文提出嵌套子空间网络(NSNs),一种新型架构范式,使单个模型在推理时能动态、精细地调整至连续的计算预算范围。核心思想是重新参数化线性层以满足嵌套子空间特性,即任意低秩函数严格包含于更高秩函数中。我们通过不确定性感知目标联合优化整个层级模型,根据任务难度自动平衡各层级贡献。实验证明,NSNs可精准应用于预训练大语言模型,并实现平滑且可预测的算力-性能边界。例如,单一适配后模型在推理时降低50%浮点运算量,仅损失5个百分点准确率。研究结果确立了NSNs作为构建下一代自适应基础模型的强大框架。

原文摘要 · Abstract (English)

Large neural networks are typically trained for a fixed computational budget, creating a rigid trade-off between performance and efficiency that is ill-suited for deployment in resource-constrained or dynamic environments. Existing approaches to this problem present a difficult choice: training a discrete collection of specialist models is computationally prohibitive, while dynamic methods like slimmable networks often lack the flexibility to be applied to large, pre-trained foundation models. In this work, we propose Nested Subspace Networks (NSNs), a novel architectural paradigm that enables a single model to be dynamically and granularly adjusted across a continuous spectrum of compute budgets at inference time. The core of our approach is to re-parameterize linear layers to satisfy a nested subspace property, such that the function computed at a given rank is a strict subspace of the function at any higher rank. We show that this entire hierarchy of models can be optimized jointly via an uncertainty-aware objective that learns to balance the contributions of different ranks based on their intrinsic difficulty. We demonstrate empirically that NSNs can be surgically applied to pre-trained LLMs and unlock a smooth and predictable compute-performance frontier. For example, a single NSN-adapted model can achieve a 50% reduction in inference FLOPs with only a 5 percentage point loss in accuracy. Our findings establish NSNs as a powerful framework for creating the next generation of adaptive foundation models.

大模型动态推理计算优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。