让模型从专家动作中反推推理过程,实现高效知识迁移。
LeAct: Learning to Reason from Expert Actions
- 通过动作反推隐式推理链,无需专家提供语言解释。
- 在德州扑克中比基线高60毫大盲注/局,机器人任务上唯一超越直接模仿的方法。
- 适用于游戏、机器人等复杂决策场景,适合训练通用推理模型。
现代推理模型依赖人类标注或强语言模型的蒸馏数据,但专家系统(如游戏引擎、经典规划器、定理证明器)可提供大量近优动作,却无伴随的自然语言推理过程。本文提出LeAct(从动作学习推理),将推理链视为潜在变量,仅凭动作信息优化学生模型:学生生成候选推理链,保留能显著提升动作恢复概率的推理。在多尺度不完全信息博弈和模拟机器人基准测试中,小规模可达到求解器数值下限;大规模下比最强基线快5倍。在约10^9个信息集的翻牌德州扑克中,胜率领先+60毫大盲注/局;机器人任务中是唯一优于直接模仿的训练方法。本工作建立了一种系统框架,使专家系统成为基础模型的新一代推理教师。
原文摘要 · Abstract (English)
Modern reasoning models depend on reasoning data, today sourced from human annotations or distilled from stronger LLMs. However, a rich and largely untapped source of supervision lies in expert systems (e.g., game engines, classical planners, theorem provers), which routinely produce near-optimal actions across diverse domains. But these experts are silent: they commit to an action without writing down the chain of thought (CoT) behind it. Recovering that CoT as natural-language reasoning would distill expert knowledge into a student that generalizes beyond the demonstrated actions. We treat it as a latent variable and study how to recover it from the action alone. Our approach, LeAct (Learning to reason from Actions), optimizes this latent variable: the student samples candidate CoTs for each expert action, and we retain those that measurably improve its own probability of recovering the action. Across imperfect-information games at multiple scales and a simulated robotics benchmark, LeAct reaches the solver's numerical floor on small enumerable games. At larger scale, it is $5\times$ closer to the solver than the strongest expert-iteration baseline. At Flop Hold'em ($\sim 10^9$ infosets), LeAct wins head-to-head by $+60$ mbb/g, and on the robotics probe it is the only training recipe that improves on direct imitation. We present a principled framework and the result: expert systems become a categorically new source of reasoning teachers for foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。