让大模型跨语言回答更一致,靠提示词就能搞定。
Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs

- 用提示词角色设定引导模型跨语言保持事实一致。
- 在Gemma 3 12B上,提示词方法整体效果最好。
- 简单干预比深度修改更稳定,适合实际应用。
尽管大语言模型(LLMs)表现出卓越的多语言流畅性,其内部知识表示仍严重偏向高资源语言,导致跨语言事实不一致——仅因提示语言不同就改变答案分布。本文探究能否在推理阶段缓解此类偏差,使英文提示的模型像在德语、西班牙语、保加利亚语提问时一样作答。我们评估四种干预策略:零样本上下文引导(角色提示)、通过对比激活添加(CAA)操控内部表征,以及基于基准事实数据和概念泛化数据训练的直接偏好优化(DPO)轻量权重调整。为评估对齐效果,我们构建了一个多语言事实数据集,并设计了一个包含文化相关问题的新泛化基准,检验干预是否可迁移至以目标语言为中心的偏好。Gemma 3 12B Instruct 实验表明,角色提示法整体最优,兼顾效果、安全性和跨域泛化能力;CAA虽能显著提升一致性,但配置敏感且有知识退化风险;基于DPO的适配器提供持久收益,但范围窄、泛化差。结果表明,跨语言不一致至少部分是选择问题,简单上下文干预可能优于更激进的方法,实现更强鲁棒性与可迁移对齐。
原文摘要 · Abstract (English)
Although Large Language Models (LLMs) demonstrate remarkable multilingual fluency, their internal knowledge representations remain disproportionately biased toward high-resource languages. This leads to cross-lingual factual inconsistency, where they shift their empirical answer distributions based solely on the prompt language. We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages (German, Spanish, Bulgarian), and evaluate four intervention strategies: zero-shot contextual steering (persona prompting), internal representation manipulation via Contrastive Activation Addition (CAA), and lightweight weight modification via Direct Preference Optimization (DPO) trained on benchmark-derived factual data as well as conceptual generalization data. To assess alignment, we curate a multilingual factual dataset alongside a novel generalization benchmark comprising culturally rooted queries to determine whether factual interventions transfer to broader target-centric preferences. Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization. While CAA yields sharp inconsistency benchmark shifts, it is configuration-sensitive and risks knowledge degradation. DPO-based adapters offer permanent, yet narrower and less transferable gains. These findings suggest that cross-lingual inconsistency is at least partly a selection problem, and that simple contextual interventions may outperform more invasive methods for robust, transferable alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。