让评分规则自动适应生成任务，提升奖励模型的准确性与鲁棒性。
SibylSense: Adaptive Rubric Learning via Memory Tuning and Adversarial Probing
- 用可调记忆库动态更新评分规则，基于验证过的评分项进行学习。
- 在两个开放生成任务中，使评分更具有区分性，下游强化学习性能显著提升。
- 适合需要自适应奖励机制的生成模型优化场景，如对话、写作生成。
为开放式生成设计对齐且鲁棒的奖励仍是强化学习后训练的关键挑战。评分规则提供结构化、可解释的监督，但规模化构建困难：专家制定的评分成本高，提示生成的评分常流于表面或不一致，固定池的判别式评分规则易饱和和漂移，导致奖励欺骗。本文提出SibylSense，一种推理时学习方法，通过可调记忆库对冻结的评分生成器进行适应性调整。记忆库通过少量示例中参考答案与候选答案的判别差距，由验证器评估并更新评分项奖励。SibylSense交替执行记忆调优与评分对抗式策略更新，生成满足评分规则的候选答案，缩小判别差距，推动评分生成器捕捉新的质量维度。在两项开放式任务上的实验表明，SibylSense生成的评分更具区分性，且优于静态与非自适应基线的下游强化学习表现。
原文摘要 · Abstract (English)
Designing aligned and robust rewards for open-ended generation remains a key barrier to RL post-training. Rubrics provide structured, interpretable supervision, but scaling rubric construction is difficult: expert rubrics are costly, prompted rubrics are often superficial or inconsistent, and fixed-pool discriminative rubrics can saturate and drift, enabling reward hacking. We present SibylSense, an inference-time learning approach that adapts a frozen rubric generator through a tunable memory bank of validated rubric items. Memory is updated via verifier-based item rewards measured by reference-candidate answer discriminative gaps from a handful of examples. SibylSense alternates memory tuning with a rubric-adversarial policy update that produces rubric-satisfying candidate answers, shrinking discriminative gaps and driving the rubric generator to capture new quality dimensions. Experiments on two open-ended tasks show that SibylSense yields more discriminative rubrics and improves downstream RL performance over static and non-adaptive baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。