解释不应是说服工具,而是帮助决策的健康摩擦力。
Creating Healthy Friction: Determining Stakeholder Requirements of Job Recommendation Explanations
- 通过真实与随机解释对比实验,评估各角色需求
- 真实解释未显著提升决策速度与准确率
- 解释的核心价值在于提供理性思考阻力,而非增强信任
招聘中信息检索的广泛应用,主要依赖于职位推荐系统(JRS),对求职者、招聘方和企业均有重大影响。因此,这类系统在近年立法中被认定为高风险,需具备可信与透明性,使相关方理解推荐原因。为满足此要求,必须明确各利益相关方的具体偏好与需求。为此,我们采用基于真实任务的混合设计用户研究(n=30),让不同角色的利益相关方基于模型解释做出决策。该混合方法评估包含两项客观指标(正确性、效率)与三项主观指标(信任、透明度、实用性),每名参与者在真实解释和随机解释条件下各测试一次。研究还结合了针对各群体适配任务的“出声思维”定性分析。结果显示,真实解释并未显著提升决策速度与准确性;但在所有利益相关方中,真实解释在感知信任、实用性和透明度上呈现非显著正向趋势。我们发现,解释更应作为支持决策的‘健康摩擦’,而非预设的‘说服工具’。
原文摘要 · Abstract (English)
The increased use of information retrieval in recruitment, primarily through job recommender systems (JRSs), can have a large impact on job seekers, recruiters, and companies. As a result, such systems have been determined to be high-risk in recent legislature. This requires JRSs to be trustworthy and transparent, allowing stakeholders to understand why specific recommendations were made. To fulfill this requirement, the stakeholders' exact preferences and needs need to be determined. To do so, we evaluated an explainable job recommender system using a realistic, task-based, mixed-design user study (n=30) in which stakeholders had to make decisions based on the model's explanations. This mixed-methods evaluation consisted of two objective metrics - correctness and efficiency, along with three subjective metrics - trust, transparency, and usefulness. These metrics were evaluated twice per participant, once using real explanations and once using random explanations. The study included a qualitative analysis following a think-aloud protocol while performing tasks adapted to each stakeholder group. We find that providing stakeholders with real explanations does not significantly improve decision-making speed and accuracy. Our results showed a non-significant trend for the real explanations to outperform the random ones on perceived trust, usefulness, and transparency of the system for all stakeholder types. We determine that stakeholders benefit more from interacting with explanations as decision support capable of providing healthy friction, rather than as previously-assumed persuasive tools.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。