提出作者归属不公平性度量,发现中心作者更易被误指
Quantifying Misattribution Unfairness in Authorship Attribution
- 引入MAUIk指标,衡量作者在非自己写作文本中被高估的频率
- 五种模型在两数据集上均显示显著不公平,中心作者风险更高
- 适合关注模型公平性与司法应用风险的读者
作者归属错误可能在现实中有深远影响。在法医场景中,仅被列为某段文本的潜在作者就可能导致不必要审查。这引发公平性问题:候选作者是否面临同等的误归风险?现有作者归属评估指标未显式考虑这一公平性。本文提出一种简单度量——误归属不公平指数(MAUIk),基于作者在非自己撰写的文本中被排进前k名的频率。利用该指标,我们量化了五种模型在两个数据集上的不公平程度。所有模型均表现出高水平的不公平,部分作者面临更高风险。进一步发现,这种不公平与模型在隐空间中对作者向量的嵌入方式相关:靠近作者群体中心的作者更易被误归属。结果表明此类模型存在潜在危害,需向使用者沟通并校准误归属风险。
原文摘要 · Abstract (English)
Authorship misattribution can have profound consequences in real life. In forensic settings simply being considered as one of the potential authors of an evidential piece of text or communication can result in undesirable scrutiny. This raises a fairness question: Is every author in the candidate pool at equal risk of misattribution? Standard evaluation measures for authorship attribution systems do not explicitly account for this notion of fairness. We introduce a simple measure, Misattribution Unfairness Index (MAUIk), which is based on how often authors are ranked in the top k for texts they did not write. Using this measure we quantify the unfairness of five models on two different datasets. All models exhibit high levels of unfairness with increased risks for some authors. Furthermore, we find that this unfairness relates to how the models embed the authors as vectors in the latent search space. In particular, we observe that the risk of misattribution is higher for authors closer to the centroid (or center) of the embedded authors in the haystack. These results indicate the potential for harm and the need for communicating with and calibrating end users on misattribution risk when building and providing such models for downstream use.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。