arXiv:2507.12143cs.CL2025-07综述被引 2

用LLM当老师、学生和评分员,评测文本理解能力。

Overview of the Sensemaking Task at the ELOQUENT 2025 Lab: LLMs as Teachers, Students and Evaluators

  • 设计三步流程:生成问题、回答问题、评分答案,模拟课堂评估。
  • 多语言测试材料下,模型答问整体尚可但受限于原文内容。
  • 评分系统易被误导,暴露LLM评卷的可信度问题,适合研究者参考。

ELOQUENT是一组共享任务,旨在为生成式语言模型提供可测试的高层次评估标准。其中,Sensemaking任务旨在评估生成模型从给定文本中“理清逻辑”的能力,流程借鉴课堂教学:(1) 教师系统生成问题,(2) 学生系统作答,(3) 评价系统评分,均严格基于输入材料。本文报告2025年版Sensemaking任务,使用7类测试材料(包括事实核查分析、教科书、讲座录音及教育视频),覆盖英语、德语、乌克兰语和捷克语。共4支队伍参与,提交2份教师、2份学生、2份评价方案。我们引入商业大模型作为教师与学生基线,并设计全自动评估流程,与轻量人工评估对比。观察发现:问题生成阶段仍缺乏有效评估策略;问答阶段模型表现总体可接受,但受限于原文的答不准问题仍普遍存在;评价阶段对抗测试显示,基于LLM-as-a-Judge的系统会错误认可混乱或错配的答案,存在严重误判风险。

原文摘要 · Abstract (English)

ELOQUENT is a set of shared tasks that aims to create easily testable high-level criteria for evaluating generative language models. Sensemaking is one such shared task. In Sensemaking, we try to assess how well generative models ``make sense out of a given text'' in three steps inspired by exams in a classroom setting: (1) Teacher systems should prepare a set of questions, (2) Student systems should answer these questions, and (3) Evaluator systems should score these answers, all adhering rather strictly to a given set of input materials. We report on the 2025 edition of Sensemaking, where we had 7 sources of test materials (fact-checking analyses of statements, textbooks, transcribed recordings of a lecture, and educational videos) spanning English, German, Ukrainian, and Czech languages. This year, 4 teams participated, providing us with 2 Teacher submissions, 2 Student submissions, and 2 Evaluator submissions. We added baselines for Teacher and Student using commercial large language model systems. We devised a fully automatic evaluation procedure, which we compare to a minimalistic manual evaluation. We were able to make some interesting observations. For the first task, the creation of questions, better evaluation strategies will still have to be devised because it is difficult to discern the quality of the various candidate question sets. In the second task, question answering, the LLMs examined overall perform acceptably, but restricting their answers to the given input texts remains problematic. In the third task, evaluation of question answers, our adversarial tests reveal that systems using the LLM-as-a-Judge paradigm erroneously rate both garbled question-answer pairs and answers to mixed-up questions as acceptable.

文本理解评估框架LLM评测多语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。