通过控制专家权重读取量，显著降低大模型合并的I/O开销。
Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Merging
- 将大模型合并建模为专家权重访问集问题，按预算选择读取块
- 实测减少高达90%的专家读取量，速度提升最高11倍
- 适用于资源受限场景的大模型高效合并，如边缘部署
权重空间模型合并通常被当作检查点间的代数运算，但在大语言模型规模下，限制因素常是必须读取的专家权重集合。我们提出MergePipe，一种预算感知的执行层，将大模型合并建模为一个‘专家访问集’问题：在共享权重坐标系下，给定合并算子和检查点族，需在显式I/O预算约束下选择读取哪些专家差分块。MergePipe对参数块进行索引，生成确定性访问计划，并通过可重现的清单执行预算合并。该计划在构造上满足预算安全，全预算时可恢复完整读取合并；对于固定系数加法型合并算子，被忽略更新的误差由被跳过的差分块范数有界。在Qwen与Llama合并工作负载中，MergePipe将专家读取I/O最多降低一个数量级，实现最高11倍加速。代表性预算扫描显示，参数偏差低至10⁻³量级，下游基准测试无单调退化。
原文摘要 · Abstract (English)
Weight-space model merging is usually formulated as an algebraic operation on checkpoints, yet at LLM scale the limiting resource is often the set of expert weights that must be read. We introduce MergePipe, a budget-aware execution layer that casts LLM merging as an \emph{expert access-set} problem: given a merge operator and a checkpoint family in a shared weight coordinate system, choose which expert delta blocks to access under an explicit I/O budget. MergePipe indexes parameter blocks, builds deterministic access plans, and executes the induced budgeted merge with replayable manifests. The plan is budget-sound by construction and recovers the full-read merge at full budget; for fixed-coefficient additive operators, the omitted-update error is bounded by the norm of omitted deltas. Across Qwen and Llama merging workloads, MergePipe reduces expert-read I/O by up to an order of magnitude and achieves up to $11\times$ speedups. Representative budget sweeps show $O(10^{-3})$ parameter deviation from full-read merges and no monotonic degradation on downstream benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。