梳理112篇论文，揭示机器学习中'专家'定义模糊的现状
What Makes An Expert? Reviewing How ML Researchers Define "Expert"
- 分析112篇论文中对'专家'的描述，发现定义普遍模糊
- 专家知识多局限于教科书式内容，依赖数据标注等单一形式
- 呼吁明确专家角色，拓展对非正式知识的认可
人类专家常参与机器学习系统的数据收集与验证、算法咨询及性能评估。然而，何为‘专家’、‘专业知识’如何界定并未明确定义。本文回顾了112篇明确提及‘专家’或‘专业知识’并涉及机器学习系统开发的学术论文，调查其对专家角色的刻画。研究发现，专业知识常未被明确定义，且非正式教育背景或职业认证外的知识极少被重视，这影响了哪些知识在机器学习发展中被认可与合法化。此外，专家知识多用于提取教科书式知识，如通过数据标注实现。本文讨论了专家参与与去技能化、专业知识的社会建构之间的关系，并指出负责任的人工智能发展需反思和具体化领域专家参与的理由，以提升可追溯性与可复现性，并拓宽被认可的专业知识范畴。
原文摘要 · Abstract (English)
Human experts are often engaged in the development of machine learning systems to collect and validate data, consult on algorithm development, and evaluate system performance. At the same time, who counts as an 'expert' and what constitutes 'expertise' is not always explicitly defined. In this work, we review 112 academic publications that explicitly reference 'expert' and 'expertise' and that describe the development of machine learning (ML) systems to survey how expertise is characterized and the role experts play. We find that expertise is often undefined and forms of knowledge outside of formal education and professional certification are rarely sought, which has implications for the kinds of knowledge that are recognized and legitimized in ML development. Moreover, we find that expert knowledge tends to be utilized in ways focused on mining textbook knowledge, such as through data annotation. We discuss the ways experts are engaged in ML development in relation to deskilling, the social construction of expertise, and implications for responsible AI development. We point to a need for reflection and specificity in justifications of domain expert engagement, both as a matter of documentation and reproducibility, as well as a matter of broadening the range of recognized expertise.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。