构建多层级评测基准,评估大模型在固废管理中的专业决策能力。
WuYuEval: A Multi-Level Benchmark for Large Language Models in Solid Waste Management

- 分基础、领域推理、专家决策三层设计,覆盖6类任务8个领域。
- 领先模型基础模块准确率达94.64%,难题仅42.50%正确。
- 强调工程约束与多目标权衡,适合研发固废专用大模型者参考。
大型语言模型(LLMs)作为技术助手日益普及,但其在固废管理(SWM)领域的专业能力难以评估,因现有基准侧重通用知识而非工程、环境与政策约束下的专业决策。我们提出WuYuEval,一个涵盖基础认知、领域推理与专家决策的多层级评测基准。经质量审核后,基准包含基础模块:4,590道封闭式多选题,覆盖六类任务与八个领域;专家模块:247道情景式开放题,涉及多目标优化、约束权衡与系统设计。专家任务采用锚定校准的LLM作为裁判评分与基于Elo的成对比较。33个模型测试显示,领先模型在基础模块准确率达94.64%,但难题准确率降至42.50%,计算、实验设计、城市规划及开放题表现更差。强调推理思维虽提升多数模型对,但增益依赖基线能力且不一致。结果表明,可见推理仅在保持单位、假设与工程约束时有效,否则易偏离关键边界。WuYuEval为开发具备专业推理链与显式约束控制的固废领域基础模型提供了评测资源与实证依据。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly used as technical assistants, but their competence in solid waste management (SWM) remains difficult to assess because existing benchmarks emphasize general knowledge rather than professional decisions under engineering, environmental, and policy constraints. We introduce WuYuEval, a multi-level benchmark for evaluating LLMs in SWM across foundational knowledge, domain reasoning, and expert decision-making. After quality auditing, WuYuEval contains a Foundation Module with 4,590 closed-ended multiple-choice questions across six task types and eight domain categories, together with an Expert Module with 247 scenario-based open-ended questions involving multi-objective optimization, constraint trade-offs, and system design. For expert tasks, we combine anchor-calibrated LLM-as-a-Judge scoring with Elo-based pairwise comparison. Across 33 LLMs, performance varied widely. The leading model reached 94.64\% accuracy on the Foundation Module, but average accuracy still fell from 84.14\% on easy questions to 42.50\% on hard questions, with lower performance concentrated in calculation, experimental design, urban planning, and open-ended expert tasks. Reasoning-oriented Thinking modes improve most matched model pairs after auditing, but the gains depend on baseline capability and are not uniformly positive. These results suggest that visible deliberation helps only when it remains anchored to units, assumptions, and engineering constraints; otherwise, it may drift from decisive answer boundaries. WuYuEval therefore provides both an evaluation resource and an empirical basis for developing SWM-oriented foundation models with professional reasoning chains and explicit constraint control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。