通过检测评估分数连续下降，自动终止奖励过优化的RLHF任务，提升资源利用率。
EvalStop: Using World Feedback to Detect and Correct Reward Overoptimization in Multi-Tenant RLHF Platforms
- 基于评估分数连续下降触发任务终止，避免奖励模型过度拟合人类偏好
- 在80% RLHF负载下实现98%精确率、99%召回率，降低1.5%误报率
- 适用于多租户平台，兼容各类调度器，无需人工干预
云端大模型微调平台日益承担强化学习人类反馈（RLHF）任务，其中学习到的奖励模型作为人类质量的代理。如Gao等（2023）所示，在持续优化压力下，该代理会偏离真实世界反馈（下游评估指标），即奖励过优化现象。现有调度器忽视这一偏差：非预见性调度器仅优化完成时间（JCT）而无质量信号；SLAQ类质量感知调度器使用训练损失（更弱的代理，会单调下降且易被操纵）；传统单任务早停需人工监控，无法释放共享GPU。我们提出EvalStop，一种可组合的调度原语：当连续k次评估分数下降时终止任务，释放GPU，保留最佳检查点，并交由任意基础调度器处理。将调度级早停视为检测问题，在离散事件仿真器中评估，工作负载混合奖励操纵与结构健康运行，真实标签对调度器不可见。在80% RLHF负载（64块GPU）下，EvalStop实现98%精度、99%召回率、1.5%假阳性率，使JCT改善9%，浪费计算减少22%（相比SRTF-Est，p<0.05）。固定进度和损失平台竞争者要么在健康运行中产生65%假阳性，要么错过超过一半的真实操纵案例。收益在所有测试基础调度器上均成立（JCT改善9-25%），且在评估噪声（标准差≤0.05时精度≥91%）和操纵比例（20%-80%范围内精度≥89%）变化下保持稳定。
原文摘要 · Abstract (English)
Cloud LLM fine-tuning platforms increasingly serve RLHF workloads, where a learned reward model is optimized as a proxy for human quality. As Gao et al. (2023) showed, this proxy diverges from world feedback (downstream eval metrics) under sustained optimization pressure, a phenomenon known as reward overoptimization. Existing platform schedulers ignore this divergence: non-clairvoyant schedulers optimize JCT without any quality signal, SLAQ-style quality-aware schedulers use training loss (a weaker proxy that drops monotonically through hacking), and classical per-job early stopping requires human monitoring and does not free shared GPUs. We propose EvalStop, a composable scheduling primitive that terminates jobs on k consecutive eval-score declines, releases GPUs, preserves the best checkpoint, and delegates to any base scheduler. We frame scheduler-level early stopping as a detection problem and evaluate it in a discrete-event simulator whose RLHF workload mixes reward-hacking and structurally healthy runs, with ground-truth labels hidden from schedulers. On RLHF-heavy workloads (80% RLHF, 64 GPUs), EvalStop achieves precision 98% / recall 99% / FPR 1.5% while improving JCT by 9% and cutting wasted compute by 22% over SRTF-Est (p<0.05). Trivial fixed-progress and loss-plateau competitors either incur 65% FPR on healthy RLHF or miss over half of true hacking cases. Gains compose across every base scheduler tested (9-25% JCT) and detection quality stays stable under eval noise (precision at least 91% at noise std <= 0.05) and hacking base rate (precision at least 89% across 20-80% hacking fractions).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。