用验证器反馈校准教师指导,提升长文本推理的准确性
Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning

- 按回合分组归一化验证器与教师信号,生成带符号差异残差
- 在五个长文本任务上,模型准确率提升至40.47和44.65
- 适合需要全局一致性推理的场景,如法律、医学问答
基于策略的蒸馏(OPD)通过强教师的逐标记指导训练学生模型。但在长上下文任务中,这种局部指导可能导致忽略跨输入分布的证据或违反全局约束。任务验证器则在响应层面评估任务完成度,可返回反映部分成功的评分。我们诊断了两个代表性长文本证据聚合任务中固定响应的匹配度问题:随着输入长度增加,轨迹级OPD得分与验证器评分的对齐度逐渐下降,表明教师与验证器存在分歧。为此,提出分组校准的基于策略蒸馏(GC-OPD)。GC-OPD在每一批次内分别归一化验证器奖励与轨迹级OPD得分,并以差值作为带符号的教师-验证器分歧残差。相对优势信用分配(RACA)根据各标记的相对OPD优势将该残差分配到对应标记,同时保留原有OPD信号。在五个长文本基准测试中,使用GC-OPD微调后,Qwen3-4B和Qwen3-8B官方检查点的平均得分分别从29.08提升至40.47,从35.12提升至44.65;而标准OPD仅达39.31和43.56。受控消融实验表明,带符号残差优于额外的OPD项或直接添加归一化验证器奖励,且RACA优于均匀分配。结果表明,分组相对残差校准可在不丢弃密集标记级指导的前提下融合验证器输出。
原文摘要 · Abstract (English)
On-policy distillation (OPD) trains a student on its own responses using dense token-level guidance from a stronger teacher. In long-context tasks, however, token-level teacher support can favor locally plausible responses that omit evidence distributed across the input or violate global task constraints. Task-specific verifiers, in contrast, evaluate task completion at the response level and may return graded rewards that reflect partial success. We diagnose this mismatch on fixed responses from two representative long-context evidence-aggregation tasks. Across longer input ranges, trajectory-level OPD scores become progressively less aligned with verifier rewards, indicating teacher-verifier disagreement. Motivated by this observation, we introduce Group-Calibrated On-Policy Distillation (GC-OPD). GC-OPD separately normalizes verifier rewards and trajectory-level OPD scores within each rollout group and uses their difference as a signed teacher-verifier disagreement residual. Relative-advantage-based credit assignment (RACA) distributes this trajectory-level residual across tokens according to their relative OPD advantages while preserving the original OPD signal. Across five long-context benchmarks, post-training with GC-OPD raises the five-benchmark averages of the official Qwen3-4B and Qwen3-8B checkpoints from 29.08 to 40.47 and from 35.12 to 44.65, respectively. Vanilla OPD reaches 39.31 and 43.56 under the same setup. Controlled ablations show that the signed residual is more effective than either an additional OPD-derived term or direct group-normalized verifier reward addition, while RACA further improves over uniform token allocation. Together, these results demonstrate that group-relative residual calibration can incorporate verifier outcomes without discarding dense token-level guidance. Code is available at https://github.com/SolereZhang/GC-OPD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。