让大模型在推理时‘多思考’,提升内部知识的推理能力。
Improving Latent Generalization Using Test-time Compute
- 用强化学习训练模型生成思维链以增强潜在泛化能力。
- 在分布内任务上显著修复了知识推理失败问题,且能泛化到新知识。
- 适合想提升模型逻辑推理能力的研究者和开发者。
语言模型具备两种知识获取机制:权重内学习(将信息编码于模型参数)与上下文学习(ICL)。尽管二者互补,权重内学习在推理内部知识时常表现不佳,体现为潜在泛化能力不足,如反转任务中的‘反向诅咒’。相比之下,上下文学习展现出极强的潜在泛化能力。为改善权重内知识的潜在泛化,现有方法依赖训练时的数据增强,但此类方法任务特定、扩展性差,且无法泛化至分布外知识。本文研究模型如何利用测试时计算(即‘思考’)来提升潜在泛化。我们采用基于正确性反馈的强化学习,训练模型生成长思维链(CoTs)以增强推理。实验表明,该‘思考’方法不仅解决了分布内知识的多数泛化失败问题,还超越了增强基线,在未进行强化学习的新知识上实现泛化。然而,在纯反转任务中,思考未能直接实现知识反转,但其生成-验证能力使模型性能远超随机水平。事实自验证的脆弱性导致其仍远低于上下文学习的表现。总体而言,测试时思考为提升语言模型潜在泛化提供了灵活且有前景的方向。
原文摘要 · Abstract (English)
Language Models (LMs) exhibit two distinct mechanisms for knowledge acquisition: in-weights learning (i.e., encoding information within the model weights) and in-context learning (ICL). Although these two modes offer complementary strengths, in-weights learning frequently struggles to facilitate deductive reasoning over the internalized knowledge. We characterize this limitation as a deficit in latent generalization, of which the reversal curse is one example. Conversely, in-context learning demonstrates highly robust latent generalization capabilities. To improve latent generalization from in-weights knowledge, prior approaches rely on train-time data augmentation, yet these techniques are task-specific, scale poorly, and fail to generalize to out-of-distribution knowledge. To overcome these shortcomings, this work studies how models can be taught to use test-time compute, or 'thinking', specifically to improve latent generalization. We use Reinforcement Learning (RL) from correctness feedback to train models to produce long chains-of-thought (CoTs) to improve latent generalization. Our experiments show that this thinking approach not only resolves many instances of latent generalization failures on in-distribution knowledge but also, unlike augmentation baselines, generalizes to new knowledge for which no RL was performed. Nevertheless, on pure reversal tasks, we find that thinking does not unlock direct knowledge inversion, but the generate-and-verify ability of thinking models enables them to get well above chance performance. The brittleness of factual self-verification means thinking models still remain well below the performance of in-context learning for this task. Overall, our results establish test-time thinking as a flexible and promising direction for improving the latent generalization of LMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。