让大模型学会主动提问,提升推理准确性和效率。
Socratic Students: Teaching Language Models to Learn by Asking Questions
- 通过任务结果反向训练模型提问策略,无需人工标注。
- 在数学和编程任务上,准确率提升最高达54.7%(绝对值)。
- 用更少对话轮次达到相同效果,适合需要深度交互的场景。
大型语言模型通常用于回答问题,但许多高风险应用(如教学辅导、临床支持)需要相反的能力:主动提问以发现信息缺失、请求澄清并推进任务解决。本文研究在推理密集型领域中,通过提问推动进展的能力。设计了一种互动协议,学生模型在有限轮次预算下与更强的教师模型交互。每轮教师回复后,以Pass@k评估学生在原任务上的表现。提出一种基于任务结果的提问优化策略(ODQS),从下游任务结果中学习提问策略。每轮采样多个候选问题,分别向教师提问并根据学生表现得分,再通过监督微调与直接偏好优化(DPO)训练学生模型,全程无需人类标注。在GSM8K、HumanEval和OpenCoder数据集上,相比基线方法显著提升性能,数学任务上Pass@5最高提升54.7%(绝对值),编码任务提升22.9%,且在三轮内达成基线水平。表明可仅凭任务结果显式训练提问能力,显著提升交互推理的准确性与效率。
原文摘要 · Abstract (English)
Large language Models (LLMs) are usually used to answer questions, but many high-stakes applications (e.g., tutoring, clinical support) require the complementary skill of asking questions: detecting missing information, requesting clarifications, and using them to solve tasks. We study this skill in reasoning-heavy domains where progress depends on inquiry rather than factual recall. We define an interactive protocol where a student model engages a stronger teacher under a small turn budget. After each teacher reply, we evaluate the student on the original task with Pass@k. We propose Outcome-Driven Question optimization Strategy (ODQS ), a training framework that learns a questioning policy from downstream task outcomes. At each turn, we sample multiple candidate questions; query the teacher with each, then score the student's resulting performance. Using these scores, we train the student via supervised fine-tuning followed by Direct Preference Optimization (DPO), without any human labels. On GSM8K, HumanEval, and OpenCoder, ODQS produces large gains over interactive baselines, boosting Pass@5 by up to 54.7% (absolute) on math and 22.9% (absolute) on coding, and matching baseline performance in three fewer turns. Thus, question asking can be explicitly trained from task outcomes, improving both accuracy and efficiency in interactive reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。