评测大模型个性化生成的鲁棒性,发现主流模型易出错,提出新方法提升准确性。
Benchmarking and Improving LLM Robustness for Personalized Generation
- 构建PERG框架与数据集,评估模型在个性化下的事实一致性
- 7B级模型错误率超20%,即使GPT-4.1也在5%情况下失准
- 提出Pref-Aligner方法,平均提升25%鲁棒性,适合需可靠个性化的场景
近年来,大语言模型(LLMs)的个性化响应受到广泛关注。现有评估主要关注回复是否符合用户偏好,但我们认为事实性同样关键。在个性化背景下,我们将模型鲁棒性定义为回复既准确又符合用户偏好。为此,我们提出了PERG框架及新数据集PERGData,评估了来自五个模型家族的十四种模型在不同提示策略下的表现。结果显示,当前模型在鲁棒性上存在明显不足:即使最强模型(GPT-4.1、LLaMA3-70B)在未个性化时曾成功的案例中,仍有5%出现不准确;而小型模型(如7B规模)错误率超过20%。进一步分析表明,鲁棒性受查询类型和用户偏好类型显著影响。为此,我们提出Pref-Aligner——一种两阶段方法,在各模型上平均提升25%的鲁棒性。本工作揭示了当前评估实践中的关键缺口,并提供了工具与指标,以支持更可靠、用户对齐的LLM部署。
原文摘要 · Abstract (English)
Recent years have witnessed a growing interest in personalizing the responses of large language models (LLMs). While existing evaluations primarily focus on whether a response aligns with a user's preferences, we argue that factuality is an equally important yet often overlooked dimension. In the context of personalization, we define a model as robust if its responses are both factually accurate and align with the user preferences. To assess this, we introduce PERG, a scalable framework for evaluating robustness in LLMs, along with a new dataset, PERGData. We evaluate fourteen models from five different model families using different prompting methods. Our findings show that current LLMs struggle with robust personalization: even the strongest models (GPT-4.1, LLaMA3-70B) fail to maintain correctness in 5% of previously successful cases without personalization, while smaller models (e.g., 7B-scale) can fail more than 20% of the time. Further analysis reveals that robustness is significantly affected by the nature of the query and the type of user preference. To mitigate these failures, we propose Pref-Aligner, a two-stage approach that improves robustness by an average of 25% across models. Our work highlights critical gaps in current evaluation practices and introduces tools and metrics to support more reliable, user-aligned LLM deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。