用强化学习提升小模型的语义理解能力，效果超过GPT-4o。
Improving the Language Understanding Capabilities of Large Language Models Using Reinforcement Learning
- 将语言理解任务转化为强化学习环境，用奖励信号优化生成过程。
- 在GLUE上平均提升6.3分，超越零样本提示38.7分。
- 适合追求高效微调、无须大量标注数据的研究者与开发者。
参数量低于140亿的指令微调大模型在自然语言理解任务上仍表现不佳，常落后于BERT-base等小型模型。受强化学习在推理任务中成功应用的启发（如DeepSeek），本文探索使用近端策略优化（PPO）提升大模型的语义理解能力。将自然语言理解建模为强化学习环境，将词元生成视为一系列动作，以与真实标签对齐的奖励信号进行优化。实验表明，PPO显著优于监督微调，在GLUE基准上平均提升6.3分；相比零样本和少样本提示，分别高出38.7分和26.1分。值得注意的是，经过PPO微调的模型在情感分析与自然语言推理任务上平均超越GPT-4o超过4%，其中在Mental Health数据集上提升7.3%，在SIGA-nli上提升10.9%。该研究揭示了一种通过重构任务为强化学习问题来高效适配大模型的新路径，仅需终端任务奖励即可实现学习，无需复杂数据标注。
原文摘要 · Abstract (English)
Instruction-fine-tuned large language models (LLMs) under 14B parameters continue to underperform on natural language understanding (NLU) tasks, often trailing smaller models like BERT-base on benchmarks such as GLUE and SuperGLUE. Motivated by the success of reinforcement learning in reasoning tasks (e.g., DeepSeek), we explore Proximal Policy Optimization (PPO) as a framework to improve the NLU capabilities of LLMs. We frame NLU as a reinforcement learning environment, treating token generation as a sequence of actions and optimizing for reward signals based on alignment with ground-truth labels. PPO consistently outperforms supervised fine-tuning, yielding an average improvement of 6.3 points on GLUE, and surpasses zero-shot and few-shot prompting by 38.7 and 26.1 points, respectively. Notably, PPO-tuned models outperform GPT-4o by over 4\% on average across sentiment and natural language inference tasks, including gains of 7.3\% on the Mental Health dataset and 10.9\% on SIGA-nli. This work highlights a promising direction for adapting LLMs to new tasks by reframing them as reinforcement learning problems, enabling learning through simple end-task rewards rather than extensive data curation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。