通过预设提示工程提升强化微调效果,让模型学会不同推理风格。
Prior Prompt Engineering for Reinforcement Fine-Tuning
- 将推理、规划等推理时提示策略转化为训练时的预设提示
- 零样本提示法在多个评测集上表现最优,尤其在AIME2024和GPQA-Diamond上领先
- 不同预设提示可让模型内化出不同行为模式,适合定制化训练
本文研究强化微调(RFT)中的预设提示工程(pPE),即在训练时于查询前添加指令以引导语言模型产生特定行为(如逐步推理)。现有研究多关注算法、奖励设计与数据构建,而预设提示的设计仍被忽视。受推理时提示工程(iPE)启发,本文将五种典型iPE策略——推理、规划、基于代码的推理、知识回忆与零样本示例利用——转化为对应的pPE方法。在Qwen2.5-7B模型上进行实验,评估其在域内与域外基准(如AIME2024、HumanEval+、GPQA-Diamond)上的表现。结果表明,所有pPE训练模型均优于对应iPE提示版本,其中零样本提示法带来最大平均性能提升,并在AIME2024和GPQA-Diamond上超越常用推理提示。通过行为分类框架进一步验证,不同pPE策略使模型表现出显著不同的行为风格。这表明pPE是强化微调中一个强大但未被充分探索的方向。
原文摘要 · Abstract (English)
This paper investigates prior prompt engineering (pPE) in the context of reinforcement fine-tuning (RFT), where language models (LMs) are incentivized to exhibit behaviors that maximize performance through reward signals. While existing RFT research has primarily focused on algorithms, reward shaping, and data curation, the design of the prior prompt--the instructions prepended to queries during training to elicit behaviors such as step-by-step reasoning--remains underexplored. We investigate whether different pPE approaches can guide LMs to internalize distinct behaviors after RFT. Inspired by inference-time prompt engineering (iPE), we translate five representative iPE strategies--reasoning, planning, code-based reasoning, knowledge recall, and null-example utilization--into corresponding pPE approaches. We experiment with Qwen2.5-7B using each of the pPE approaches, then evaluate performance on in-domain and out-of-domain benchmarks (e.g., AIME2024, HumanEval+, and GPQA-Diamond). Our results show that all pPE-trained models surpass their iPE-prompted counterparts, with the null-example pPE approach achieving the largest average performance gain and the highest improvement on AIME2024 and GPQA-Diamond, surpassing the commonly used reasoning approach. Furthermore, by adapting a behavior-classification framework, we demonstrate that different pPE strategies instill distinct behavioral styles in the resulting models. These findings position pPE as a powerful yet understudied axis for RFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。