针对对话游戏小模型的错误,提出诊断引导的三步训练法。
Acquire, Repair, Preserve: A Diagnosis-Guided Post-Training Recipe for Small-Model Dialogue Game Agents
- 通过监督微调获取广泛游戏参与能力
- 用局部偏好对修复特定游戏中的机械错误,得分提升超27点
- 适合关注对话系统纠错与可迁移性研究者
互动对话游戏测试了静态基准通常忽略的能力:模型需在多轮中保持状态、理解反馈,并在变化约束下做出有效动作。我们在LM Playschool挑战中使用一个20亿参数的开源模型进行研究,发现许多失败不仅是知识不足,还包括局部决策失误:重复猜测、无效动作以及违反刚看到的反馈。这些诊断催生了一种三步训练方案:首先通过监督微调(SFT)获得广泛的游戏参与能力;其次在单一对话游戏家族内,利用逐轮偏好对修复可验证的机械错误;最后保护超出这些游戏的通用能力。在官方最终评测中,我们的提交将公开ClemScore从10.67提升至38.92,封闭域内分数从13.41升至41.17,而整体静态性能基本不变(44.14 vs. 44.24)。跨域ClemScore仍低(7.88),主要增益集中在目标家族的未见变体。结果表明,广义SFT贡献了大部分能力提升;当故障检测精准时,逐轮监督有效,且迁移主要集中于同家族内部。
原文摘要 · Abstract (English)
Interactive dialogue games test a capability that static benchmarks largely leave implicit: a model must carry state across turns, interpret feedback, and choose valid actions under changing constraints. We study this setting in the LM Playschool Challenge with a 2B open-weight model, and find that many failures are not only broad knowledge failures but also local decision failures: repeated guesses, malformed actions, and violations of feedback that the model has just seen. These diagnostics motivate a training recipe organized around three steps: acquire broad game participation through supervised fine-tuning, repair mechanically verifiable failures within one targeted dialogue-game family using turn-local preference pairs, and preserve general capabilities beyond these dialogue games. In the official final evaluation, our submission improves public clemscore from 10.67 to 38.92 and closed in-domain score from 13.41 to 41.17, while approximately preserving aggregate static performance (44.14 vs. 44.24 for the baseline). Out-of-domain clemscore remains low at 7.88, with the largest gains concentrated in unseen variants of the targeted family. Our results suggest that broad SFT brings most of the model's capability improvement; turn-local supervision can be effective when failure detection is precise, with observed transfer concentrated primarily within-family.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。