arXiv:2504.19898cs.CL2025-04被引 8

用强化学习提升大模型分类能力,不靠推理步骤反而更准。

GenCLS++: Pushing the Boundaries of Generative Classification in LLMs Through Comprehensive SFT and RL Studies Across Diverse Datasets

  • 联合优化监督微调与强化学习,系统探索五类提示策略。
  • 在七大数据集上平均提升3.46%,公开数据集达4.00%。
  • 发现分类任务无需推理过程,对实际应用有指导意义。

文本分类是机器学习中的基础任务,在诸多领域至关重要。随着大语言模型(LLMs)的快速扩展,尤其是通过强化学习(RL)训练,对更强大判别器的需求日益增长。传统判别方法将文本映射到标签,忽略了大模型内在的生成优势。生成式分类通过直接提示模型输出标签来弥补这一缺陷。然而,现有研究多仅依赖简单监督微调(SFT),很少探究训练与推理提示的交互关系,也无工作系统性地结合强化学习与统一的SFT、RL及推理时提示策略。本文提出GenCLS++框架,联合优化SFT与强化学习,并系统探索五类高层策略维度:上下文学习变体、类别定义方式、显式不确定性标签、语义无关数值标签以及基于困惑度的解码策略。经过SFT“策略预热”后,采用简单规则奖励进行强化学习,取得显著增益。在七个数据集上,相对于朴素的SFT基线,平均准确率提升3.46%;在公开数据集上提升达4.00%。值得注意的是,不同于需要推理过程的任务,我们发现分类任务在不使用显式推理步骤的情况下表现更优。这些关于显式推理作用的洞察为未来大模型应用提供了重要指导。

原文摘要 · Abstract (English)

As a fundamental task in machine learning, text classification plays a crucial role in many areas. With the rapid scaling of Large Language Models (LLMs), particularly through reinforcement learning (RL), there is a growing need for more capable discriminators. Consequently, advances in classification are becoming increasingly vital for enhancing the overall capabilities of LLMs. Traditional discriminative methods map text to labels but overlook LLMs' intrinsic generative strengths. Generative classification addresses this by prompting the model to directly output labels. However, existing studies still rely on simple SFT alone, seldom probing the interplay between training and inference prompts, and no work has systematically leveraged RL for generative text classifiers and unified SFT, RL, and inference-time prompting in one framework. We bridge this gap with GenCLS++, a framework that jointly optimizes SFT and RL while systematically exploring five high-level strategy dimensions-in-context learning variants, category definitions, explicit uncertainty labels, semantically irrelevant numeric labels, and perplexity-based decoding-during both training and inference. After an SFT "policy warm-up," we apply RL with a simple rule-based reward, yielding sizable extra gains. Across seven datasets, GenCLS++ achieves an average accuracy improvement of 3.46% relative to the naive SFT baseline; on public datasets, this improvement rises to 4.00%. Notably, unlike reasoning-intensive tasks that benefit from explicit thinking processes, we find that classification tasks perform better without such reasoning steps. These insights into the role of explicit reasoning provide valuable guidance for future LLM applications.

生成分类强化学习大模型提示工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。