模型自动生成题目和答案，不用外部数据就能提升解题能力。
Self-Questioning Language Models
- 用自生成问题进行自我训练，通过强化学习让模型互问互答。
- 在代数、编程等任务上，无需标注数据即可提升性能。
- 适合想用纯模型能力提升推理的开发者或研究者。
大型语言模型能否仅通过自生成问题和答案来提升能力？我们提出自提问语言模型（SQLM）：一种非对称自博弈框架，其中提议者根据主题生成问题，求解者尝试解答。两者均通过强化学习训练。提议者获得奖励若问题难度适中；求解者依据多数投票得分，作为无真实答案时的正确性代理。对于编程任务，提议者可生成单元测试用于验证。我们在三个基准上评估：三位数乘法、OMEGA基准的代数题、Codeforces的编程题。模型通过持续生成更复杂的问题并尝试解决，可在不依赖任何标注数据集的情况下提升下游任务表现。
原文摘要 · Abstract (English)
Can large language models improve without external data -- by generating their own questions and answers? We hypothesize that a pre-trained language model can improve its reasoning skills given only a single prompt specifying the topic (e.g., algebra word problems) and asking the model to generate its own questions. To do this, we propose Self-Questioning Language Models (SQLM): an asymmetric self-play framework where a proposer is given the topic and generates a question for a solver, who tries to answer it. Both the proposer and solver are trained via reinforcement learning. The proposer receives a reward if the problem is not too easy or too difficult, and the solver receives a reward based on majority voting, a proxy for correctness in the absence of ground-truth answers. For coding, the proposer can instead generate unit tests which are used for verification. We study this asymmetric self-play framework on three benchmarks: three-digit multiplication, algebra problems from the OMEGA benchmark, and programming problems from Codeforces. By continually generating more interesting problems and attempting to solve them, language models can improve on downstream benchmarks without access to any curated training datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。