让大模型学会用弱模型能跟上的方式推理,提升可解释性。
Tandem Training for Language Models
- 用弱模型随机接管强模型的推理过程,强制其输出可传递的步骤。
- 在GSM8K任务中准确率保持高位,同时减少术语使用。
- 适合需要人类或弱模型监督的长期人机协作场景。
随着语言模型能力不断提升,其决策与推理过程可能变得对弱代理和人类难以理解,削弱可解释性与监管能力。为此,我们提出以‘手递手鲁棒性’定义可理解性:强模型的解决方案若能在任意阶段交由弱模型继续执行而不失败,则视为可理解。基于此,我们引入协同训练(tandem training)——一种强化学习范式,其中在推理轨迹中随机且间歇地从一个冻结的弱模型采样令牌,而非强模型自身。只有当强模型的动作和推理路径能被弱模型延续时,该轨迹才成功。因此,优化标准强化学习目标会隐式激励正确性与可理解性双重目标。在GSM8K数学推理任务中,协同训练使模型可靠地避免使用专业术语,并调整语言风格以适应弱合作方,同时维持高任务准确率。结果表明,这为构建仍可被弱代理审计的AI系统提供了有前景的路径,对人机协作和多智能体通信具有重要意义。
原文摘要 · Abstract (English)
As language models continue to rapidly improve, we can expect their actions and reasoning to become difficult or impossible for weaker agents and humans to follow, undermining interpretability and oversight. With an eye on long-term futures, we pursue methods that encourage models to produce solutions that remain intelligible to weaker collaborators. We formalize intelligibility as handoff robustness: a strong model's solution is intelligible to a weaker model if randomly handing off control to the weaker model along the solution path does not cause failure. Building on this criterion, we introduce tandem training for language models, a reinforcement learning (RL) paradigm in which rollout tokens are intermittently and randomly sampled from a frozen weak model rather than the strong model being trained. Because rollouts succeed only when the strong model's actions and reasoning process can be continued by the weak model -- when the two can co-construct a successful solution -- optimizing standard RL objectives with tandem training implicitly incentivizes both correctness and intelligibility. In the GSM8K math reasoning task, tandem training reliably teaches models to abandon jargon and adapt their language to weaker partners while keeping task accuracy high. Our results demonstrate a promising route to building AI systems that remain auditable by weaker agents, with implications for human--AI collaboration and multi-agent communication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。