评估大模型升级时适配器迁移的决策效果，帮企业省算力少标注。
UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists
- 构建跨版本纵向基准，模拟真实模型迭代中的适配器迁移决策。
- 部分任务升级后性能提升显著，但多数任务在训练噪声范围内波动。
- 轻量级探测可预测适配器可移植性，支持低开销迁移决策。
组织为开源大模型维护特定任务的适配器，每次新基模型发布都需决定：保留旧适配器、迁移、从保留行为中恢复或重新训练。现有迁移研究仅评估孤立模型对，未涵盖真实模型发布序列中的决策。我们提出UpgradeBench，一个以决策为中心的纵向基准，覆盖四个连续的Qwen发布版本、一个持续预训练检查点、六个任务和两种模型规模，并扩展了具有已知训练谱系的OLMo检查点。该基准分离三个核心问题：新检查点是否提升固定配方重训练适配器的性能；专长资产能否跨版本迁移；可用的恢复资源有哪些。实验发现，升级收益在不同任务-规模-发布周期中差异显著：部分重训练基线表现提升，另一些则处于训练噪声内，稳定性从文本到SQL的不足一发布周期至意图分类超十四个月不等。直接复制适配器与架构或模型家族无关：在OLMo上，保留率从460亿令牌持续预训练时的0.88–0.99降至2.9万亿令牌时归零；退火与模型拼接无额外损害，可移植性随持续预训练距离衰减。在保留输入数据条件下，教师重标注可恢复目标基模型适配器，无需新标注，但计算节省不保证。在33次升级事件中模拟固定决策策略，平均质量损失仅0.37个百分点，且无行为退化，仅需完整重训练1/3的计算与标注成本。256个提示的轻量级CKA探针可在八个模型对上以Spearman相关系数0.74预测跨版本适配器可移植性。我们公开每例预测、成本日志、划分清单及评估代码。
原文摘要 · Abstract (English)
Organizations maintain task-specific adapters for open-weight language models, and each new base-model release forces a migration decision: retain existing specialists, port adapters, refresh from preserved behavior, or retrain. Prior transfer work evaluates isolated model pairs, without studying these choices across real model release sequences. We present UpgradeBench, a decision-driven longitudinal benchmark covering four consecutive Qwen releases, one continuation checkpoint, six tasks, and two model sizes, augmented by OLMo checkpoints with known training lineage. The benchmark disentangles three core questions: whether a new checkpoint improves fixed-recipe retrained specialist performance, whether specialization assets transfer across versions, and what recovery resources are usable. We observe upgrade gains differ across task-scale-release episodes: some retrained baselines improve while others stay within training noise, with durability ranging from under one release interval for text-to-SQL to over fourteen months for intent classification. Direct adapter copying depends neither on architecture nor model family: on OLMo, retention drops from 0.88-0.99 at 46B-token continued pretraining to zero at 2.9T tokens; annealing and model souping introduce no extra harm, with portability decaying with continued-pretraining distance. Given preserved input data, teacher relabeling recovers target-base specialists without fresh gold annotations, though compute savings are not guaranteed. Simulating a fixed decision policy over 33 upgrade episodes yields 0.37pp mean quality regret with zero behavioral regressions at one-third the compute and label cost of full retraining. A lightweight CKA probe over 256 prompts predicts cross-version adapter portability (Spearman 0.74 across eight model pairs). We release per-example predictions, cost logs, split manifests, and evaluation code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。