用自然语言摘要评估大模型能力，更直观全面。
Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summaries
- 用自然语言生成模型表现摘要，提升可读性。
- 通过三项标准验证摘要质量，区分模型优劣。
- 无需人工标注即可自动生成，适合开发者参考。
大语言模型（LLM）发展迅速且动态变化，传统量化评测难以准确反映其真实能力。本文提出「报告卡」机制，即针对特定技能或主题的、人类可理解的自然语言摘要，用于描述模型行为。我们构建了基于三重标准的评估框架：特异性（区分不同模型的能力）、忠实性（准确反映模型实际表现）和可解释性（对人类清晰相关）。此外，我们设计了一种无需人工标注的迭代生成算法，并通过消融实验验证其有效性。在多个主流大模型上的实验证明，报告卡能提供超越传统基准的洞察，有助于实现更可解释、更全面的LLM评估。
原文摘要 · Abstract (English)
The rapid development and dynamic nature of large language models (LLMs) make it difficult for conventional quantitative benchmarks to accurately assess their capabilities. We propose report cards, which are human-interpretable, natural language summaries of model behavior for specific skills or topics. We develop a framework to evaluate report cards based on three criteria: specificity (ability to distinguish between models), faithfulness (accurate representation of model capabilities), and interpretability (clarity and relevance to humans). We also propose an iterative algorithm for generating report cards without human supervision and explore its efficacy by ablating various design choices. Through experimentation with popular LLMs, we demonstrate that report cards provide insights beyond traditional benchmarks and can help address the need for a more interpretable and holistic evaluation of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。