用启发式方法自动去偏，低成本提升大模型公平性。
A Heuristic Perspective on Debiasing Language Models
- 用简单模板生成提示词，暴露模型偏见
- 通过最小化预测分布差异，实现去偏微调
- 无需人工标注，适配多文化场景
语言模型在预训练过程中常习得各类偏见，并在交互中表现出来，可能造成社会危害。现有方法多依赖反事实增强或表示投影，但计算开销大，难以扩展到大模型，且需人工标注数据，适用范围受限于特定文化与偏见类别。为此，我们提出HEIMAT——一种启发式自动去偏框架。该框架包含两步：首先使用简单模板构造启发式提示，揭示模型偏见并生成上下文提示；其次通过最小化这些上下文提示上的预测分布间詹森-香农散度，对模型进行去偏微调。大量实验表明，HEIMAT能有效缓解不同文化背景下的偏见，同时保持模型自然语言理解能力。
原文摘要 · Abstract (English)
Language models (LMs) often acquire various biases during pre-training and may express them in interactions, potentially causing social harm. Existing methods often rely on counterfactual augmentation or representation projection. These strategies remain limited in practice due to their high computational costs and difficulty in scaling to larger models. Additionally, many of these strategies require manual data annotation, narrowing their scope to specific cultures and bias categories. To overcome these limitations, we propose HEIMAT, a HEurIstic-style autoMATic debiasing framework for LMs. HEIMAT consists of two main steps: bias disclosure and debiasing fine-tuning. In the first step, it uses simple templates to construct heuristic prompts, which are applied to reveal model biases and generate corresponding context prompts. In the second step, it fine-tunes the model by minimizing the Jensen-Shannon divergence of predictions on these context prompts to reduce bias. Extensive experiments show that HEIMAT effectively mitigates bias in different cultures while maintaining the model's natural language understanding (NLU) performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。