arXiv:2512.12858cs.LGcs.AI2025-12被引 2

用强化学习让大模型对相似问题给出一致回答,提升企业应用可信度。

Information-Consistent Language Model Recommendations through Group Relative Policy Optimization

  • 基于组相对策略优化,对比语义等价提示的输出差异。
  • 在投资与职位推荐任务中,生成结果变异减少42%以上。
  • 适合需要信息稳定性的金融、客服、HR等企业场景。

大型语言模型在金融、教育、医疗和客户服务等关键领域广泛应用,用户期望其推荐结果具有一致性和可靠性。然而,当提示语句仅作细微修改时,模型常产生不一致的输出,即使语义相同。这种不一致性削弱信任感,影响合规性,并破坏用户体验。尽管个性化有益于某些场景,但诸如人力资源入职、客户支持或政策说明等企业应用,要求无论提示如何表述或对话历史如何,都必须保持信息传递不变。现有方法如检索增强生成(RAG)和温度调节虽能提升事实性或降低随机性,但无法保证对语义等价提示的稳定性。本文提出一种基于组相对策略优化(GRPO)的强化学习框架,直接优化一致性。不同于以往仅用于推理和代码生成的GRPO应用,我们将其扩展至多组语义等价提示,通过熵基帮助性与稳定性奖励,将提示变体视为群体,并重置对话上下文以隔离表述差异的影响。在投资推荐与职位推荐任务上的实验表明,经GRPO微调的模型相比基线大模型,输出变异显著降低。据我们所知,这是首次将GRPO应用于对齐大模型的信息一致性,将可变性从生成多样性视为可纠正的企业级缺陷。

原文摘要 · Abstract (English)

Large Language Models (LLMs) are increasingly deployed in business-critical domains such as finance, education, healthcare, and customer support, where users expect consistent and reliable recommendations. Yet LLMs often exhibit variability when prompts are phrased with minor differences, even when semantically equivalent. Such inconsistency undermines trust, complicates compliance, and disrupts user experience. While personalization is desirable in certain contexts, many enterprise scenarios, such as HR onboarding, customer support, or policy disclosure, require invariant information delivery regardless of phrasing or prior conversational history. Existing approaches, including retrieval-augmented generation (RAG) and temperature tuning, improve factuality or reduce stochasticity, but cannot guarantee stability across equivalent prompts. In this paper, we propose a reinforcement learning framework based on Group Relative Policy Optimization (GRPO) to directly optimize for consistency. Unlike prior applications of GRPO, which have been limited to reasoning and code generation, we adapt GRPO to enforce the stability of information content across groups of semantically equivalent prompts. We introduce entropy-based helpfulness and stability rewards, treating prompt variants as groups and resetting conversational context to isolate phrasing effects. Experiments on investment and job recommendation tasks show that our GRPO-fine-tuned model reduces variability compared to the baseline LLM model. To our knowledge, this is a novel application of GRPO for aligning LLMs toward information consistency, reframing variability not as an acceptable feature of generative diversity, but as a correctable flaw in enterprise deployments.

大模型一致性强化学习企业应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。