不用微调，用三步提示让大模型裁判准确率提升14个百分点。
RTLC -- Research, Teach-to-Learn, Critique: A three-stage prompting paradigm inspired by the Feynman Learning Technique that lifts LLM-as-judge accuracy on JudgeBench with no fine-tuning
- 模仿费曼学习法设计三阶段提示：研究、教学、批判，激发模型多角度思考。
- 在JudgeBench上使Claude 3.7 Sonnet的判别准确率从64.6%提升至78.6%。
- 无需训练或外部工具，适合希望提升模型评估能力的研究者使用。
LLM作为裁判已成为开放生成任务的标准评估工具，但在公开的JudgeBench基准上，即使强指令微调的裁判模型在客观正确性配对题上也仅略高于随机水平。本文提出RTLC——一种受费曼学习法启发的三阶段提示范式（研究→教学→批判），可在不进行微调、检索或使用外部工具的情况下，将单一黑箱大模型转化为多思维集成裁判。第一阶段采用固定教学框架，引导模型模拟学习过程；第二阶段在温度0.4下生成10个独立判断结果；第三阶段以自身为批评者，交叉对比候选结果与原始问题，输出最终判据（温度0）。在包含350个难题的JudgeBench-GPT上，Claude 3.7 Sonnet的配对准确率从64.6%（单次提示）提升至78.6%（批判-10），绝对提升14.0个百分点。该方法优于10样本自洽多数投票（77.7%）和首份答案（74.0%）。消融实验表明，教学阶段贡献9.4个百分点，10样本取平均贡献3.7个百分点，显式批判贡献0.9个百分点。本文还分析了成本-精度权衡，误差预算分布，并验证了其与事后评分校准的正交性，二者可实现乘法级增益。
原文摘要 · Abstract (English)
LLM-as-a-judge is now the default measurement instrument for open-ended generation, but on the public JudgeBench benchmark even strong instruction-tuned judges barely scrape past random on objective-correctness pairwise items. We introduce RTLC, a three-stage prompting recipe -- Research, Teach-to-Learn, Critique -- that promotes a single black-box LLM into an ensemble-of-thought judge with no fine-tuning, retrieval, or external tools. Stage 1 wraps the input in a fixed pedagogical scaffold porting the Feynman Learning Technique (study $\to$ teach $\to$ find gaps $\to$ simplify) into LLM prompting. Stage 2 draws N=10 independent candidate verdicts at temperature 0.4. Stage 3 acts as its own critic, cross-comparing the candidate set against the original question to emit one critiqued verdict at temperature 0. On JudgeBench-GPT (350 hard pairwise items), Claude 3.7 Sonnet's pairwise accuracy climbs from 64.6% (single-shot vanilla prompt) to 78.6% (RTLC critique-of-10) -- an absolute 14.0-percentage-point gain. RTLC also beats N=10 self-consistency majority voting (77.7%) and a zero-shot first candidate (74.0%). A clean three-step ablation attributes +9.4 pp to the Teach-to-Learn scaffold, +3.7 pp to N=10 marginalisation, and +0.9 pp to explicit critique. We discuss the cost-accuracy frontier (RTLC sits above self-consistency at every working point), the error-budget breakdown across the four JudgeBench categories (knowledge, reasoning, math, coding), and how RTLC composes orthogonally with post-hoc judge-score calibration, with the two interventions compounding multiplicatively in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。