让大模型自动学规则,提升评估通用性与准确性
Learned-Rule-Augmented Large Language Model Evaluators
- 用MCTS从数据中自动提取评分规则,避免人工设计
- 提出链式规则和强化学习训练,使模型更好应用规则
- 在多种任务上验证,通用评估能力更强
大语言模型(LLMs)广泛用于自然语言生成(NLG)任务的评估,但在更广泛的评估场景中应用受限。现有方法因依赖昂贵的人工设计评价准则而难以泛化,且常与标注数据及模型理解不一致。为此,我们提出一种规则增强的评估范式:首先,通过LLM辅助的蒙特卡洛树搜索(MCTS)自动从数据中提取评分规则,缓解可扩展性问题并提升与数据的一致性;其次,提出两种策略使LLM有效应用所学规则:(1) 链式规则(CoR),引导模型遵循提炼出的规则;(2) 通过强化学习训练规则增强的LLM评估器(RuAE),进一步弥合规则与模型推理之间的差距。在多种任务上的大量实验表明,该方法在不同评估场景中均具备良好有效性和泛化能力。
原文摘要 · Abstract (English)
Large language models (LLMs) are predominantly used as evaluators for natural language generation (NLG) tasks, but their application to broader evaluation scenarios remains limited. In this work, we explore the potential of LLMs as general evaluators across diverse tasks. Although LLM-based evaluators have made progress in different areas, existing methods struggle to generalize due to their reliance on costly, human-designed evaluation principles, which are often misaligned with both annotated data and LLMs' understanding.To address these challenges, we propose a rule-augmented evaluation paradigm. First, we introduce a rule distillation method that automatically extracts scoring rules from data using an LLM-assisted Monte Carlo Tree Search (MCTS), alleviating scalability issues and improving alignment with data. Second, to enable LLMs to effectively apply the learned rules, we propose two strategies: (1) Chain-of-Rule (CoR), which guides LLM to follow distilled rules, and (2) training a rule-augmented LLM evaluator (RuAE) via reinforcement learning, further bridging the gap between rules and LLMs' reasoning. Extensive experiments on diverse tasks demonstrate the effectiveness and generalizability of our approach across various evaluation scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。