提出评估生成内容公平性的新框架,明确目标分布应如何合理设定。
Who Should Be Generated? Justifying Demographic Targets in Open-Ended Generation

- 构建四重承诺框架,系统化定义输出侧人口属性的目标分配逻辑。
- 实测发现模型生成结果与地理来源目标差异显著,最大偏离达0.606。
- 适用于需对生成内容进行公平性审计的研究者与从业者。
公平性评估不仅关注模型生成了什么,还应关注其输出应以何种标准为参照。当提示词为“美国的一位首席执行官”时,模型自行决定人口属性实现。现有群体公平性定义假设敏感属性在输入端已知,而生成审计则考察输出端的人口构成,但其对比目标通常未经论证即被采用。核心问题在于目标分布应如何确定。本文针对无指定人口属性的生成任务,形式化缺失目标问题,并将目标构建分解为四个承诺:评估对象、先验合理性、分配方式与可操作性。在此框架下,我们承认在公开世界使用场景中,地理归属作为先验是合理的;职业先验则需独立辩护,如劳动力构成一致性。在AP-Bench上实例化该框架,发现生成结果与地理衍生目标间存在显著偏差,范围为0.508至0.606(0-1尺度)。若用同类别比较器替代地理目标,在保持生成和测量不变的前提下,模型特定的平均绝对单元级JSD₂变化范围为0.279至0.355。因此,目标构建并非公平性评估的前置步骤,而是其组成部分。我们提供的不是通用目标,而是一个使目标正当性显式化的框架。
原文摘要 · Abstract (English)
Fairness evaluation concerns not only what a model produces, but also what its outputs ought to be compared against. When a model generates "a CEO in the United States," the prompt leaves demographic realization to the model. Existing group fairness definitions assume that sensitive attributes are given on the input side. Generative audits instead examine output-side demographic composition, yet the targets they compare it against are typically supplied rather than justified. The upstream question is what the target distribution should be. We formalize this missing-target problem for demographic-value-unspecified generation and decompose target construction into four commitments: the evaluative object, prior admissibility, allocation, and operationalization. In this framework, we admit the geographic prior under a geographic-membership interpretation for the declared public-world use. The occupational prior, under an incumbency interpretation, requires an independently defended objective such as workforce-composition fidelity. Instantiating this construction in AP-Bench, we find substantial distribution divergence from geography-derived targets, ranging from 0.508 to 0.606 on a 0-to-1 scale. Replacing each geography-derived target with an equal-category comparator, while holding generations and measurement fixed, produces model-specific mean absolute cell-level $\mathrm{JSD}_2$ changes ranging from 0.279 to 0.355. Target construction is therefore not a preliminary to fairness evaluation but a component of it. What we supply is not a universal target, but a framework that makes explicit the justification required before a distribution can serve as a fairness standard.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。