用可收缩基底动态组合，高效适配MoE模型。
Dynamic Subspace Composition: Efficient Adaptation via Contractive Basis Expansion
- 通过状态相关稀疏基底扩展，构建上下文依赖权重
- 参数量降至O(M d)，内存传输降低至O(Kd)
- 适合需要高效微调的大型模型部署
Mixture of Experts (MoE) 模型虽具高容量，但常面临表征坍塌与梯度不稳问题。本文提出动态子空间组合（DSC）框架，通过状态相关的稀疏基底银行对上下文依赖权重进行近似。形式上，DSC将权重更新建模为星形域内的残差轨迹，并采用幅度门控单纯形插值确保在恒等映射处的连续性。与标准 MoE-LoRA（需检索独立的秩-r 矩阵，参数复杂度 O(M rd)）不同，DSC从解耦的单位范数基向量中构造秩-K 的组合近似，将参数复杂度降至 O(M d)，内存流量降至 O(Kd)。帧理论正则化与谱约束提供了动态更新的严格最坏情况边界。代码已开源：https://github.com/VladimerKhasia/DSC
原文摘要 · Abstract (English)
Mixture of Experts (MoE) models scale capacity but often suffer from representation collapse and gradient instability. We propose Dynamic Subspace Composition (DSC), a framework that approximates context-dependent weights via a state-dependent, sparse expansion of a shared basis bank. Formally, DSC models the weight update as a residual trajectory within a Star- Shaped Domain, employing a Magnitude-Gated Simplex Interpolation to ensure continuity at the identity. Unlike standard Mixture-of-LoRAs, which incurs O(M rd) parameter complexity by retrieving independent rank-r matrices, DSC constructs a compositional rank-K approximation from decoupled unit-norm basis vectors. This reduces parameter complexity to O(M d) and memory traffic to O(Kd), while Frame-Theoretic regularization and spectral constraints provide rigorous worst-case bounds on the dynamic update. The code is available at https://github. com/VladimerKhasia/DSC
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。