为大模型实用价值评估建立系统框架，超越单纯任务表现。
From Performance to Purpose: A Sociotechnical Taxonomy for Evaluating Large Language Model Utility
- 构建四维评估体系：性能、交互、运营、治理，覆盖真实场景需求。
- 提出可量化的评估指标库，支持跨场景模型选择与比较。
- 配套在线工具，动态链接评估维度与实际可用指标。
随着大语言模型在离散任务上的表现持续提升，它们正被整合进日益复杂多样的现实系统中。然而，仅靠任务成功无法判定模型在实践中的适用性。在高风险应用环境中，大模型的有效性受多种社会技术因素驱动，远超传统性能指标。尽管已有大量度量方法涵盖这些考量，但缺乏统一组织，难以实现一致评估。为此，我们提出语言模型实用价值分类体系（LUX），从性能、交互、运营和治理四个领域构建层次化框架，每个领域包含主题对齐的维度与组件，均基于可量化的指标，支持模型选型与应用场景的精准匹配。同时提供外部动态网页工具，将每个组件关联至相关度量指标库，便于实际评估应用。
原文摘要 · Abstract (English)
As large language models (LLMs) continue to improve at completing discrete tasks, they are being integrated into increasingly complex and diverse real-world systems. However, task-level success alone does not establish a model's fit for use in practice. In applied, high-stakes settings, LLM effectiveness is driven by a wider array of sociotechnical determinants that extend beyond conventional performance measures. Although a growing set of metrics capture many of these considerations, they are rarely organized in a way that supports consistent evaluation, leaving no unified taxonomy for assessing and comparing LLM utility across use cases. To address this gap, we introduce the Language Model Utility Taxonomy (LUX), a comprehensive framework that structures utility evaluation across four domains: performance, interaction, operations, and governance. Within each domain, LUX is organized hierarchically into thematically aligned dimensions and components, each grounded in metrics that enable quantitative comparison and alignment of model selection with intended use. In addition, an external dynamic web tool is provided to support exploration of the framework by connecting each component to a repository of relevant metrics (factors) for applied evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。