用小模型自动构建更鲁棒的LLM知识评估基准，成本更低质量不降。
SAGE: Scalable Automated Robustness Augmentation for LLM Knowledge Evaluation
- 用微调小模型生成问题变体，再用人工标注规则验证真伪。
- 在HellaSwag上生成的基准与人工标注版质量相当，成本大幅降低。
- 方法可推广至MMLU等任务，无需针对新数据集重新训练。
大型语言模型在标准知识评估基准上表现优异，但面对同一知识的不同表述形式时仍显脆弱。因此亟需对现有评估基准进行鲁棒性增强。然而，当前基于LLM的生成-验证流程因变体生成效率低、验证不可靠而难以扩展。本文提出SAGE（可扩展的鲁棒性评估基准生成框架），利用微调的小模型实现高效增强。SAGE包含VariantQual（基于规则的人工标注验证器）和VariantGen（经监督微调并用VariantQual作为奖励模型进行强化学习优化的生成器）。在HellaSwag上的实验表明，SAGE以显著更低的成本构建了大规模鲁棒性增强基准，其质量接近人工标注的HellaSwag-Pro；且微调模型在无需特定基准微调的情况下，可泛化到MMLU任务。
原文摘要 · Abstract (English)
Large Language Models (LLMs) achieve strong performance on standard knowledge evaluation benchmarks, yet recent work shows that their knowledge capabilities remain brittle under question variants that test the same knowledge in different forms. Robustness augmentation of existing knowledge evaluation benchmarks is therefore necessary, but current LLM-assisted generate-then-verify pipelines are costly and difficult to scale due to low-yield variant generation and unreliable variant verification. We propose SAGE (Scalable Automated Generation of Robustness BEnchmarks), a framework for scalable robustness augmentation of knowledge evaluation benchmarks using fine-tuned smaller models. SAGE consists of VariantQual, a rubric-based verifier trained on human-labeled seed data, and VariantGen, a variant generator initialized with supervised fine-tuning and further optimized with reinforcement learning using VariantQual as the reward model. Experiments on HellaSwag show that SAGE constructs a large-scale robustness-augmented benchmark with quality comparable to the human-annotated HellaSwag-Pro at substantially lower cost, while the fine-tuned models further generalize to MMLU without benchmark-specific fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。