arXiv:2501.02334cs.CLcs.AI2025-01被引 4

用生成式AI评分需收集更全面的证据,确保结果可信可靠。

Validity Arguments For Constructed Response Scoring Using Generative Artificial Intelligence Applications

  • 对比三类评分系统:人工、基于特征的AI、生成式AI
  • 生成式AI需更多验证证据,因缺乏透明性和一致性问题
  • 适合关注AI评分可靠性与评估方法的教育技术研究者

大型语言模型和生成式人工智能的快速发展,使它们在高风险考试中的应用日益可能。将生成式AI用于开放回答题评分尤为吸引人,因为它减少了传统AI评分中手工设计特征的工作量,甚至可能表现更优。本文旨在阐明特征基础方法与生成式AI在开放回答评分系统中的差异,并提出支持生成式AI评分结果使用与解释的有效性证据收集最佳实践。我们比较了人工评分、基于自然语言处理的特征型AI评分系统与生成式AI评分所需的有效性证据。由于生成式AI缺乏透明性及一致性等独特问题,其所需证据比特征型NLP评分系统更为广泛。标准化测试的开放回答数据展示了不同评分系统有效性证据的收集过程,凸显了在缺乏人工评分时,结合多个AI评分源的协同评分方法能更全面覆盖测评构念。

原文摘要 · Abstract (English)

The rapid advancements in large language models and generative artificial intelligence (AI) capabilities are making their broad application in the high-stakes testing context more likely. Use of generative AI in the scoring of constructed responses is particularly appealing because it reduces the effort required for handcrafting features in traditional AI scoring and might even outperform those methods. The purpose of this paper is to highlight the differences in the feature-based and generative AI applications in constructed response scoring systems and propose a set of best practices for the collection of validity evidence to support the use and interpretation of constructed response scores from scoring systems using generative AI. We compare the validity evidence needed in scoring systems using human ratings, feature-based natural language processing AI scoring engines, and generative AI. The evidence needed in the generative AI context is more extensive than in the feature-based NLP scoring context because of the lack of transparency and other concerns unique to generative AI such as consistency. Constructed response score data from standardized tests demonstrate the collection of validity evidence for different types of scoring systems and highlights the numerous complexities and considerations when making a validity argument for these scores. In addition, we discuss how the evaluation of AI scores might include a consideration of how a contributory scoring approach combining multiple AI scores (from different sources) will cover more of the construct in the absence of human ratings.

AI评分有效性证据生成式AI教育评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。