系统化评估大模型，兼顾实际场景与伦理考量。
The Science of Evaluating Foundation Models
- 构建适配具体场景的评估框架，提升针对性。
- 提供检查清单与模板，确保评估可重复、可落地。
- 梳理大模型评估前沿进展，聚焦真实应用需求。
大模型的涌现能力已彻底改变自然语言处理领域，但其评估面临规模大、能力广、应用场景多等挑战。现有研究多关注单一指标或任务表现，未能整合多样使用场景与更广泛的伦理和运营考量。本文聚焦三大核心：(1) 通过结构化框架形式化评估流程，适配具体应用场景；(2) 提供可操作的工具与模板，保障评估的完整性、可复现性与实用性；(3) 系统回顾近期工作，重点梳理大模型评估的最新进展，强调面向真实世界的落地应用。
原文摘要 · Abstract (English)
The emergent phenomena of large foundation models have revolutionized natural language processing. However, evaluating these models presents significant challenges due to their size, capabilities, and deployment across diverse applications. Existing literature often focuses on individual aspects, such as benchmark performance or specific tasks, but fails to provide a cohesive process that integrates the nuances of diverse use cases with broader ethical and operational considerations. This work focuses on three key aspects: (1) Formalizing the Evaluation Process by providing a structured framework tailored to specific use-case contexts, (2) Offering Actionable Tools and Frameworks such as checklists and templates to ensure thorough, reproducible, and practical evaluations, and (3) Surveying Recent Work with a targeted review of advancements in LLM evaluation, emphasizing real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。