不修改模型权重,在生成时用评分机制实时消除性别、种族等偏见。
Decoding-Time Debiasing via Process Reward Models: From Controlled Fill-in to Open-Ended Generation

- 用独立评分模型在生成阶段筛选更公平的词元,不碰原始模型参数。
- 序列式修正方案使偏见得分提升最高达0.40,同时保持甚至提升语言流畅性。
- 适用于小模型和大模型,仅对敏感词触发轻量门控,计算开销接近2倍。
大型语言模型会从训练数据中习得社会偏见,并在下游应用中强化性别、种族、宗教、残障、年龄及社会经济地位等刻板印象。现有修复方法(如重新训练或基于人类反馈微调)成本高、需访问模型权重,且可能损害其他任务性能。本文另辟蹊径:在解码阶段进行去偏,将去偏视为无需修改模型权重的候选词元结构化搜索。引入独立的流程评分模型(PRM)对每个候选词元在公平性与流畅性上打分。设计三种渐进复杂度的方案(Best-of-N选择、序列式批评与修订、宪法式自审计),并在涵盖八类偏见的200个双语提示基准(英文与乌尔都语)上评估四个模型(GPT-4o-mini、Llama 3.2 3B、Gemma 3 4B、Qwen 2.5 3B)。序列式去偏效果最佳,平均偏见得分较基线提升最高达+0.40,同时维持甚至改善流畅性。进一步将三种方案扩展至开放式生成,每一步动态去偏,并引入轻量级偏见防护门,仅在潜在偏见词处触发,使整体开销近似2倍(对校准良好的模型)。一项分离生成器与裁判器成本的形式化开销指标表明,原生实现下Best-of-N在生成器端几乎无额外开销。作为强商业模型锚点的GPT-4o-mini证实该框架随模型能力提升而增强;三款开源小模型则揭示当前小型LLM仍存挑战。
原文摘要 · Abstract (English)
Large language models pick up social biases from the data they are trained on and carry those biases into downstream applications, often reinforcing stereotypes around gender, race, religion, disability, age, and socioeconomic status. The standard fixes (retraining on curated data or fine-tuning with human feedback) are expensive, need access to model weights, and risk degrading the model on other tasks. In this paper we take a different route: we debias the model at decoding time, treating bias mitigation as a structured search over candidate tokens without ever touching model weights. A separate Process Reward Model (PRM) acts as a judge, scoring each candidate for both fairness and fluency. We design three schemes of increasing sophistication (Best-of-N selection, Sequential critique-and-revise, and Constitutional self-audit) and evaluate them on four models (GPT-4o-mini, Llama 3.2 3B, Gemma 3 4B, Qwen 2.5 3B) across a 200-prompt bilingual benchmark in English and Urdu covering eight bias categories. Sequential debiasing proves the most effective, raising mean bias scores by up to +0.40 over baseline while preserving (and sometimes improving) fluency. We then extend all three schemes to open-ended generation, where each token is debiased on the fly, and introduce a lightweight Bias Guard gate that fires only on potentially biased words, keeping overhead near 2x for well-calibrated models. A formal overhead metric that separates generator cost from judge cost reveals that Best-of-N is effectively free on the generator side in a native implementation. GPT-4o-mini, included as a strong proprietary anchor, confirms that the framework scales with model capability; the three open-weight models show where current small-scale LLMs still struggle.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。