arXiv:2606.00400cs.LG2026-06被引 2

用小模型动态学回放策略,大模型直接复用,防遗忘更高效。

Dynamic Proxy-Mixing: Transferring Replay Controllers from Small to Large Models for Continual Instruction Tuning

论文配图:Dynamic Proxy-Mixing: Transferring Replay Controllers from Small to Large Models for Continual Instruction Tuning
图 1 · 摘自论文原文
  • 在小模型上训练动态回放控制器,根据验证损失变化实时调整回放比例。
  • 在5个连续指令微调任务中,准确率提升3.4点,遗忘减少3.5点,安全得分提高5.8点。
  • 适合需要持续学习且资源受限的场景,部署无泄露、不依赖具体架构。

持续指令微调会逐步侵蚀模型已学能力与对齐行为。回放是常用缓解手段,但固定回放比例受限于当前领域、训练阶段及历史行为脆弱性。本文提出PROXYMIX框架:在小代理模型上学习动态回放控制器,并将其冻结后迁移到大目标模型。控制器仅依赖归一化验证损失及其时间动态构建状态,生成当前任务与可用回放缓冲区的掩码混合。核心假设为“遗忘镜像”——不同规模模型的任务脆弱性排序保持一致,尽管绝对损失值不同。该假设经实证验证后用于跨规模迁移。在LLaMA-3-8B上,五次持续指令微调序列中,相比最强非溯源基线,平均准确率提升3.4点,最终遗忘降低3.5点,安全得分提高5.8点,策略学习成本仅为溯源强化学习的1/50。框架接口层面无泄漏、架构无关,同时识别出代理假设失效场景,揭示部署局限。

原文摘要 · Abstract (English)

Continual instruction tuning updates a language model through a sequence of new domains, yet each update can progressively erode previously learned capabilities and alignment behavior. Replay is the standard mitigation, but fixed replay ratios are inherently limited because the optimal mixture varies with the current domain, the training stage, and the evolving vulnerability of prior behaviors. We propose PROX-YMIX, a framework that learns a dynamic replay controller on a small proxy model and transfers the frozen controller to a larger target. The controller never observes future tasks and constructs its state from normalized validation losses and their temporal dynamics, producing a masked mixture over the current task and accessible replay buffers. Our core empirical hypothesis is forgetting mirroring: task vulnerability rankings remain largely consistent across model scales even when absolute loss magnitudes differ. We validate this assumption empirically before transferring controllers across scales. On LLaMA-3-8B across five continual instruction tuning sequences, PROXYMIX improves average accuracy by 3.4 points, reduces final forgetting by 3.5 points, and raises safety score by 5.8 points over the strongest non-oracle baseline, at roughly 50x lower policy learning cost than Oracle Target RL. The framework is leakage free and architecture independent at the interface level, and we also identify settings where the proxy assumption breaks down, highlighting limitations for robust deployment.

持续学习回放机制模型迁移指令微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。