用强化学习提升大模型在噪声数据下的二分类准确率
BinaryPPO: Efficient Policy Optimization for Binary Classification
- 将二分类转为奖励最大化问题，用置信度加权奖励函数优化策略
- 在8个基准上提升40-60个百分点，最高达99%准确率
- 适合标签噪声多、样本不均衡的现实场景，无需在线交互
监督微调（SFT）是毒性检测、事实验证和因果推断等二分类任务的标准方法，但在存在标签噪声、类别不平衡或弱监督的现实场景中表现不佳。我们提出BinaryPPO，一种基于离线强化学习的大语言模型框架，将二分类重构为奖励最大化问题。该方法采用改进的近端策略优化（PPO），结合置信度加权奖励函数，对不确定或错误预测施加惩罚，使模型能在静态数据集上学习稳健决策策略，无需在线交互。在八个领域特定基准和多种架构的模型上，BinaryPPO相比监督基线提升40-60个百分点准确率，最高达99%。我们深入分析了奖励塑造、优势缩放与策略稳定性的作用。结果表明，基于置信度的奖励设计为二分类提供了比SFT更鲁棒的替代方案。代码已开源：https://github.com/psyonp/BinaryPPO。
原文摘要 · Abstract (English)
Supervised fine-tuning (SFT) is the standard approach for binary classification tasks such as toxicity detection, factuality verification, and causal inference. However, SFT often performs poorly in real-world settings with label noise, class imbalance, or sparse supervision. We introduce BinaryPPO, an offline reinforcement learning large language model (LLM) framework that reformulates binary classification as a reward maximization problem. Our method leverages a variant of Proximal Policy Optimization (PPO) with a confidence-weighted reward function that penalizes uncertain or incorrect predictions, enabling the model to learn robust decision policies from static datasets without online interaction. Across eight domain-specific benchmarks and multiple models with differing architectures, BinaryPPO improves accuracy by 40-60 percentage points, reaching up to 99%, substantially outperforming supervised baselines. We provide an in-depth analysis of the role of reward shaping, advantage scaling, and policy stability in enabling this improvement. Overall, we demonstrate that confidence-based reward design provides a robust alternative to SFT for binary classification. Our code is available at https://github.com/psyonp/BinaryPPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。