用大模型多角色模拟评估公平与伦理课程，发现隐藏的课程设计漏洞。
A Justice Lens on Fairness and Ethics Courses in Computing Education: LLM-Assisted Multi-Perspective and Thematic Evaluation
- 通过大模型模拟四种角色进行课程大纲评分
- 24份课程大纲中发现不同角色关注点差异显著
- 为人工智能伦理课程设计提供可操作改进方向
课程大纲决定了教学基调与预期，影响师生的学习体验。在涉及人工智能、机器学习及算法设计中的公平与伦理课程中，必须明确如何应对实现公正结果的障碍。这些预期应具有包容性、透明性，并促进批判性思维。大纲分析有助于评估课程的内容覆盖、深度、实践方式与期望。然而，人工评估耗时且易不一致。为此，我们开发了基于正义视角的评分量表，并利用大语言模型（LLM）对大纲进行多角色模拟评审。我们从教师、系主任、机构评审员和外部评审者四个视角评估了24份课程大纲，并让大模型识别跨课程的主题趋势。结果显示，多视角评估能揭示角色特异性优先事项，帮助发现聚焦公平与伦理的AI/ML相关课程中隐藏的课程设计空白。这些洞察为改进此类课程中公平、伦理与正义内容的设计与实施提供了具体建议。
原文摘要 · Abstract (English)
Course syllabi set the tone and expectations for courses, shaping the learning experience for both students and instructors. In computing courses, especially those addressing fairness and ethics in artificial intelligence (AI), machine learning (ML), and algorithmic design, it is imperative that we understand how approaches to navigating barriers to fair outcomes are being addressed.These expectations should be inclusive, transparent, and grounded in promoting critical thinking. Syllabus analysis offers a way to evaluate the coverage, depth, practices, and expectations within a course. Manual syllabus evaluation, however, is time-consuming and prone to inconsistency. To address this, we developed a justice-oriented scoring rubric and asked a large language model (LLM) to review syllabi through a multi-perspective role simulation. Using this rubric, we evaluated 24 syllabi from four perspectives: instructor, departmental chair, institutional reviewer, and external evaluator. We also prompted the LLM to identify thematic trends across the courses. Findings show that multiperspective evaluation aids us in noting nuanced, role-specific priorities, leveraging them to fill hidden gaps in curricula design of AI/ML and related computing courses focused on fairness and ethics. These insights offer concrete directions for improving the design and delivery of fairness, ethics, and justice content in such courses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。