让批评更有效:根据改写结果反推批评是否真有帮助
Co-Evolving Actor-Conditioned Critics for Non-Verifiable Generation

- 用四要素联合评估批评有效性,判断其是否针对真实问题
- 8B模型经此训练后超越120B零样本模型表现
- 批评与生成模型共同进化,适配不断变化的生成能力
自然语言批评为缺乏确定性验证器的非可验证生成任务提供了超越标量奖励的监督。在批评引导的优化中,批评者对初始回应给出反馈,生成者据此修改。但最终修改质量无法反映批评是否真正有用:能力强的生成者可能无需遵循反馈即改进,而有效的反馈也可能因生成者无法执行而失效。本文将批评视为面向特定生成者的修订指导,其有效性取决于反馈是否帮助目标生成者解决预期弱点。提出TAIScore(针对性可操作改进评分),综合评估指令、初始响应、批评和修改结果,判断批评是否针对真实缺陷、生成者是否采纳反馈、预期方面是否改善。利用该评分通过GRPO训练专属批评者,并以批评引导的改写构建DPO偏好对供生成者学习,形成批评者与生成者协同演化的闭环。实验表明,使用TAIScore训练的8B批评者优于零样本120B批评者,以及仅基于结果或仅基于批评信号训练的批评者。进一步的共演化机制提升了性能,说明有效批评监督应随生成者能力变化而自适应。
原文摘要 · Abstract (English)
Natural-language critiques provide supervision beyond scalar rewards for non-verifiable generation, which lacks deterministic verifiers. In critique-guided refinement, a critic gives feedback on an initial response and an actor revises it. However, final revision quality does not reveal whether the critique was actually useful: a capable actor may improve without following the feedback, while valid feedback may fail if the actor cannot execute it. We frame critique as actor-conditioned revision guidance, where usefulness depends on whether the feedback helps the target actor address the intended weakness. We introduce TAIScore (Targeted Actionable Improvement Score), a reward that evaluates the instruction, initial response, critique, and revision together, assessing whether the critique targets a real weakness, whether the actor follows it, and whether the intended aspect improves. We use this reward to train an actor-tailored critic with GRPO, and use critique-guided refinements to construct DPO preference pairs for the actor, forming a co-evolving critic-actor loop where the critic adapts to the actor's changing capability. Experiments show that an 8B critic trained with TAIScore outperforms both a zero-shot 120B critic and critics trained with outcome-only or critique-only reward signals. Co-evolving the critic and actor further improves performance, suggesting that effective critique supervision should adapt as the actor changes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。