arXiv:2501.05727cs.CLcs.AI2025-01中稿 · COLM被引 9

让大模型自己练评改能力,无需人工标注。

Self-Evolving Critique Abilities in Large Language Models

  • 用自生成数据训练模型自我优化评改能力。
  • 评改准确率提升10.0%,错误识别F1提高19.0%。
  • 适合想提升模型自省能力的研究者。

尽管大型语言模型(LLMs)表现优异,但在人类难以评估或模型可能超越人类的任务中,其反馈能力仍面临挑战。本文提出SCRIT(Self-evolving CRITic)框架,通过自生成数据训练模型自我演化评改能力。为解决生成数据质量低的问题,引入对比式评改机制,在数据合成中使用参考解以增强对关键概念的理解,并采用自验证方案保障数据质量。最终模型在推理时无需参考解。基于Qwen2.5-72B-Instruct实现,该方法在涵盖数学与科学推理的多个基准上均表现优异:评改准确率相对提升10.0%,错误识别F1-score相对提升19.0%。分析表明,性能随数据与模型规模正向增长,且可通过多轮迭代实现持续改进。

原文摘要 · Abstract (English)

Despite their remarkable performance, Large Language Models (LLMs) face a critical challenge: providing feedback for tasks where human evaluation is difficult or where LLMs potentially outperform humans. In such scenarios, leveraging the critique ability of LLMs themselves - identifying and correcting flaws - shows considerable promise. This paper explores enhancing critique abilities of LLMs, noting that current approaches rely on human annotations or more powerful models, leaving the challenge of improving critique abilities without external supervision unresolved. We introduce SCRIT (Self-evolving CRITic), a framework that trains LLMs with self-generated data to evolve their critique abilities. To address the low quality of naively generated data, we propose a contrastive-critic approach that uses reference solutions during data synthesis to enhance the model's understanding of key concepts, and incorporates a self-validation scheme to ensure data quality. The final trained model operates without any reference solutions at inference time. Implemented with Qwen2.5-72B-Instruct, a leading LLM, SCRIT demonstrates consistent improvements across a wide range of benchmarks spanning both mathematical and scientific reasoning: achieving a 10.0\% relative gain in critique-correction accuracy and a 19.0\% relative improvement in error identification F1-score. Our analysis reveals that SCRIT's performance scales positively with data and model size and enables continuous improvement through multi-round iterations.

大模型自我进化评改能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。