提出条件化经验复用方法,避免无效训练浪费算力。
Knowing When Not to Reuse: Conditional Experience Transfer in Autonomous LLM Post-Training
- 根据上下文条件判断是否可复用过往训练经验
- 在相同算力下比基线模型提升最终性能
- 适合自动化大模型持续训练的场景
大语言模型具备广泛能力,但适应不断演进的领域、工具和需求常需重复后训练。自主系统通过提出更新、训练候选并利用评估反馈选择后续方案来自动化部分流程。随着证据积累,核心问题浮现:后续训练改变父模型后,哪些过往更新证据仍可操作?更新效果依赖于父模型、数据与训练阶段。将过往成功视为无条件许可会浪费计算资源,若子模型被采纳,还可能破坏后续训练轨迹。本文提出条件化经验复用问题,并引入边界校准干预复用(BCIT)方法,在权重变化前授权经验复用。BCIT将观测效果绑定至其源上下文,检查适用条件,对存在命名硬冲突的候选进行否决,并在需要时通过有限训练试验获取当前状态证据。完全训练后的候选仍遵循统一采纳规则,仅可观测事件扩展记忆。在一个40亿参数模型跨金融推理、文本转SQL和函数调用任务的适配实验中,候选更新在不同情境下表现出异质的目标与保留效果。在匹配候选、证据与算力条件下,BCIT授权更少有害更新,且达到更高同等预算下的最终模型质量。结果支持将经验授权视为自主后训练中的独立问题。
原文摘要 · Abstract (English)
Large language models offer broad capabilities, but adapting them to evolving domains, tools, and requirements often entails repeated post-training. Autonomous systems automate parts of this process by proposing updates, training candidates, and using evaluation feedback to select subsequent proposals. As evidence accumulates, a central problem emerges: which past update evidence remains actionable after subsequent training has changed the parent model? An update's effect depends on its parent, data, and training stage. Treating past success as context-free permission can waste compute. If the resulting child is promoted, it can also degrade the subsequent training trajectory. We formulate this problem as conditional experience transfer and introduce Boundary-Calibrated Intervention Transfer (BCIT), a method that authorizes experience reuse before weight-changing training. BCIT binds an observed effect to its source context, checks applicability conditions, vetoes candidates with named hard conflicts, and obtains current-state evidence through a bounded training trial when needed. Fully trained candidates still face a shared adoption rule, and only observed events extend memory. On one 4B model adapted across finance reasoning, text-to-SQL, and function calling, candidate updates exhibit heterogeneous target and retention effects across the evaluated contexts. Under matched candidates, evidence, and compute, BCIT authorizes fewer harmful updates and attains higher equal-budget final-model quality than the evaluated alternatives. These results support treating experience authorization as a distinct problem in autonomous post-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。