用强化学习让大模型自动优化提问,激活隐藏推理能力。
One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinement

- 训练一个精炼器通过强化学习重写问题为逻辑分解结构。
- 在多个模型和基准上提升1.7%至7.2%,平均优于基线2.1%。
- 只需少量训练即可适配未见模型,适合部署推理优化场景。
大型语言模型常因人类提问模糊与机器所需结构化逻辑之间的分布不匹配,无法激活其隐含推理能力。现有对齐方法或需对每模型单独微调($O(N)$成本),或依赖静态提示,难以应对查询层面的结构复杂性。本文提出ReQueR(Reinforcement Query Refinement)——一种将推理激发视为推理时对齐任务的模块化框架。通过强化学习训练专用精炼器策略,将原始问题重写为显式逻辑分解,以冻结的LLM作为环境。基于教育心理学中的近侧发展区理论,引入自适应求解者层级机制,动态匹配环境难度与精炼器能力,稳定训练过程。ReQueR在多种架构与基准上实现1.7%–7.2%的一致绝对提升,平均优于强基线2.1%。关键在于,它提供了一种‘一驱多’的推理激发范式:仅需在少量模型上训练的单一精炼器,即可有效解锁多样未知模型的推理能力。代码已开源:https://github.com/newera-xiao/ReQueR。
原文摘要 · Abstract (English)
Large Language Models (LLMs) often fail to utilize their latent reasoning capabilities due to a distributional mismatch between ambiguous human inquiries and the structured logic required for machine activation. Existing alignment methods either incur prohibitive $O(N)$ costs by fine-tuning each model individually or rely on static prompts that fail to resolve query-level structural complexity. In this paper, we propose ReQueR (\textbf{Re}inforcement \textbf{Que}ry \textbf{R}efinement), a modular framework that treats reasoning elicitation as an inference-time alignment task. We train a specialized Refiner policy via Reinforcement Learning to rewrite raw queries into explicit logical decompositions, treating frozen LLMs as the environment. Rooted in the classical Zone of Proximal Development from educational psychology, we introduce the Adaptive Solver Hierarchy, a curriculum mechanism that stabilizes training by dynamically aligning environmental difficulty with the Refiner's evolving competence. ReQueR yields consistent absolute gains of 1.7\%--7.2\% across diverse architectures and benchmarks, outperforming strong baselines by 2.1\% on average. Crucially, it provides a promising paradigm for one-to-many inference-time reasoning elicitation, enabling a single Refiner trained on a small set of models to effectively unlock reasoning in diverse unseen models. Code is available at https://github.com/newera-xiao/ReQueR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。