4-bit微缩大模型适配器合并时,保持编码不变以避免精度损失。
Scale-QLoRA: Code-Invariant Adapter Merging for Native 4-bit Microscaling LLMs
- 仅调整每块缩放参数,在部署网格上训练并冻结编码平面。
- 合并后模型在四模型四任务中零精度损失,代码平面完全保留。
- 支持快速回滚、去重和125倍加速的仅缩放任务切换,适合生产部署。
将LoRA适配器合并至基础模型是标准部署流程,可消除运行时开销。但在原生4比特微缩检查点(NVFP4、MXFP4)上,此操作不再免费:合并需通过量化器重推导离散E2M1码平面(约占文件90%字节),导致部署产物绑定特定量化格式,后续任何编码操作都可能破坏其状态。若直接合并,会因与已有网格对齐而删除适应性,精度下降高达39个百分点。Scale-QLoRA则仅调整每块的原始缩放字段,在部署网格上训练并冻结所有E2M1码。在此固定格式、网格、块布局和码平面下,合并为位级恒等变换,实现代码不变性。在四个模型和四项任务中,Scale-QLoRA与融合感知的QAT-LoRA均实现无精度损失,二者结构差异在于:QAT-LoRA通过量化器重推码平面,而Scale-QLoRA完整保留。该差异带来生命周期成本——最近邻实现间误差约1个百分点,极端规则不匹配可使权重空间产物降至约0%,报告为敏感性上限而非部署频率。保留码平面还消除了训练中的权重空间直通估计器(8B稠密模型每步降低3.9倍计算量),并支持精确回滚、码平面去重及约125倍加速的仅缩放任务切换。
原文摘要 · Abstract (English)
Merging a LoRA adapter into its base model is standard deployment practice: it removes the runtime adapter's per-forward overhead and leaves a single standalone checkpoint any serving stack can load. On a native 4-bit microscaling checkpoint (NVFP4, MXFP4) that step stops being free. The merged weights must be written back through a quantizer, which re-derives the checkpoint's discrete E2M1 code plane (roughly 90% of the artifact's bytes), so the deployed artifact becomes coupled to one quantization convention, and every later code-touching event in its lifecycle can move it. Done naively the step is worse than fragile: it deletes the adaptation, by up to 39 pp, because against an already-on-grid base the reconstruction optimum is that base. Scale-QLoRA instead adapts only the native per-block scale field, trains those scales on the deployment grid, and freezes every E2M1 code. Within a fixed native format, scale grid, block layout and code plane, merging is then a bit-exact identity and the merged artifact is code-invariant. Across four models and four tasks, Scale-QLoRA and merge-aware QAT-LoRA are both accuracy-lossless, so we claim no accuracy ordering between them; they differ structurally, in that QAT-LoRA re-derives the code plane through a quantizer while Scale-QLoRA preserves it exactly. That difference is what the lifecycle prices: nearest-rounding implementations disagree by about a point on the measured task, and more extreme rule mismatches can drive the weight-space artifact to ~0%, which we report as a sensitivity bound rather than a deployment frequency. Preserving the code plane also drops the weight-space straight-through estimator from training (3.9x per step on the dense 8B model) and enables exact rollback, code-plane deduplication, and a ~125x faster scale-only task swap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。