让大模型生成更符合真实分布的文本,避免性别种族偏见。
Controlling Distributional Bias in Multi-Round LLM Generation via KL-Optimized Fine-Tuning
- 用KL散度控制隐变量令牌分布,结合语义一致性优化。
- 在6个数据集上显著优于基线,精准控制职业相关属性分布。
- 适合关注大模型公平性与可控生成的研究者使用。
现实世界本就充满随机性,但大语言模型(LLMs)通常仅在单轮推理中与固定标准对比评估。本文转向分布对齐:考察模型在多次提示下能否生成符合目标分布的输出,如反映真实统计或均匀分布。我们以职业语境中的性别、种族和情感属性来定义分布对齐。实证分析显示,现成的LLMs及标准对齐方法(如提示工程和直接偏好优化)无法可靠控制输出分布。为此,我们提出一种新微调框架,结合引导令牌校准与语义对齐。引入混合目标函数,利用Kullback-Leibler散度锚定潜在引导令牌的概率质量,同时通过Kahneman-Tversky优化将其绑定至语义一致的响应。在六个不同数据集上的实验表明,该方法显著优于基线,在属性生成任务中实现精确的分布控制。
原文摘要 · Abstract (English)
While the real world is inherently stochastic, Large Language Models (LLMs) are predominantly evaluated on single-round inference against fixed ground truths. In this work, we shift the lens to distribution alignment: assessing whether LLMs, when prompted repeatedly, can generate outputs that adhere to a desired target distribution, e.g. reflecting real-world statistics or a uniform distribution. We formulate distribution alignment using the attributes of gender, race, and sentiment within occupational contexts. Our empirical analysis reveals that off-the-shelf LLMs and standard alignment techniques, including prompt engineering and Direct Preference Optimization, fail to reliably control output distributions. To bridge this gap, we propose a novel fine-tuning framework that couples Steering Token Calibration with Semantic Alignment. We introduce a hybrid objective function combining Kullback-Leibler divergence to anchor the probability mass of latent steering tokens and Kahneman-Tversky Optimization to bind these tokens to semantically consistent responses. Experiments across six diverse datasets demonstrate that our approach significantly outperforms baselines, achieving precise distributional control in attribute generation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。