arXiv:2509.12108cs.CL2025-09EMNLP

用猜-思-答框架融合监督与强化学习,提升文本分类效率和性能

GTA: Supervised-Guided Reinforcement Learning for Text Classification with Large Language Models

  • 模型先猜答案,再反思修正,用交叉熵和强化奖励联合优化
  • 在4个基准上收敛速度超纯RL,性能超过纯SFT和纯RL基线
  • 适合追求高效高精度文本分类的工程师和研究者

在自然语言处理任务中,纯强化学习微调方法常因探索效率低导致收敛慢;而监督微调虽训练高效,但性能上限有限且理论基础较弱。为解决效率与能力之间的权衡,本文提出猜-思-答(GTA)框架,将监督微调的效率与强化学习的能力提升统一于一个训练范式。GTA通过让模型先生成初步猜测(以交叉熵损失优化),再反思该猜测并生成最终答案,利用强化学习奖励同时塑造最终输出和整个GTA结构的格式。为缓解两种训练信号间的梯度冲突,采用损失掩码和梯度约束。在四个文本分类基准上的实证结果表明,GTA显著加速收敛,同时超越独立的SFT与RL基线。

原文摘要 · Abstract (English)

In natural language processing tasks, pure reinforcement learning (RL) fine-tuning methods often suffer from inefficient exploration and slow convergence; while supervised fine-tuning (SFT) methods, although efficient in training, have limited performance ceiling and less solid theoretical foundation compared to RL. To address efficiency-capability trade-off, we propose the Guess-Think-Answer (GTA) framework that combines the efficiency of SFT with the capability gains of RL in a unified training paradigm. GTA works by having the model first produce a provisional guess (optimized via cross-entropy loss), then reflect on this guess before generating the final answer, with RL rewards shaping both the final output and the format of the entire GTA structure. This hybrid approach achieves both faster convergence than pure RL and higher performance ceiling than pure SFT. To mitigate gradient conflicts between the two training signals, we employ loss masking and gradient constraints. Empirical results on four text classification benchmarks demonstrate that GTA substantially accelerates convergence while outperforming both standalone SFT and RL baselines.

文本分类强化学习大模型微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。