用人类偏好信号优化文本分类，让模型更懂样本间相对好坏。
ClaHF: A Human Feedback-inspired Reinforcement Learning Framework for Improving Classification Tasks
- 构建候选预测与排序关系，将标签转化为可优化的偏好信号。
- 在8个任务上提升分类准确率和置信度校准效果，适配多种语言模型。
- 无需额外人工标注，直接利用现有标签生成反馈信号。
文本分类模型通常通过监督微调（SFT）训练，但SFT仅基于逐样本标签进行行为克隆，难以捕捉样本间的相对偏好关系，限制了模型对决策边界的学习和预测置信度的校准。本文提出ClaHF，一种受人类反馈启发的强化学习框架，将偏好建模与强化学习优化融入分类流程，无需额外人工标注。与仅依赖逐样本监督的方法不同，ClaHF同时构建多个候选预测及其相对排序关系，并在奖励模型中联合建模最优预测（Top-1）与非最优候选间的排序。该设计将传统标签监督转化为可直接用于策略优化的偏好信号。我们在涵盖三类场景的8个分类任务上进行了系统评估，结果表明ClaHF在多种语言模型上均持续提升了分类性能与置信度校准效果。数据与代码见https://anonymous.4open.science/r/ClaHF。
原文摘要 · Abstract (English)
Text classification models are typically trained via supervised fine-tuning (SFT). However, SFT essentially performs behavior cloning from instance-wise labels and thus fails to adequately capture relative preference relations among samples, which limits the model's ability to shape decision boundaries and calibrate predictive confidence. In this paper, we propose ClaHF, a human feedback-inspired reinforcement learning (RL) framework for text classification that integrates preference modeling and RL optimization into the classification pipeline without requiring additional human annotations. Unlike prior work that relies solely on instance-wise supervision, ClaHF constructs multiple candidate predictions together with their relative ranking relations, and jointly models the Top-1 preference and the ordering among non-optimal candidates within a reward model (RM). This design converts conventional label supervision into preference signals that are directly applicable to policy optimization. We conduct systematic evaluations on eight classification tasks spanning three categories of scenarios. Results demonstrate that ClaHF consistently improves both classification performance and confidence calibration across diverse language models (LMs). The data and code are available at https://anonymous.4open.science/r/ClaHF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。