让大模型学会推断用户隐性需求并防御性思考，提升个性化交互可靠性。
Aligning Deep Implicit Preferences by Learning to Reason Defensively
- 构建批判性推理框架，将对齐转化为可解释的推理过程。
- 在3000个查询上验证，显著提升对用户深层偏好的发现能力。
- 适合需要高鲁棒性个性化交互的研究者与开发者使用。
个性化对齐对于使大语言模型有效开展以用户为中心的交互至关重要。然而，现有方法面临双重挑战：难以推断用户的深层隐性偏好（包括未明说的目标、语义上下文和风险容忍度），且缺乏应对现实世界模糊性的防御性推理能力。这种认知差距导致回应肤浅、脆弱且短视。为此，我们提出批判驱动推理对齐（CDRA），将对齐从单一奖励匹配任务重构为结构化推理过程。首先，为弥合偏好推断缺口，我们引入DeepPref基准数据集，该数据集包含20个主题下的3000组偏好-查询对，通过模拟多维度认知委员会生成带批判标注的推理链，以解构查询语义并揭示潜在风险。其次，为植入防御性推理，我们提出个性化生成式过程奖励模型（Pers-GenPRM），将其奖励建模视为个性化推理任务，先生成批判链评估响应与用户偏好的一致性，再基于此推理输出最终评分。最终，这一可解释的结构化奖励信号通过批判驱动策略对齐（Critique-Driven Policy Alignment）算法，融合数值与自然语言反馈，指导策略模型进行过程级在线强化学习。实验表明，CDRA在发现并匹配用户真实偏好方面表现卓越，同时具备稳健推理能力。代码与数据集已开源。
原文摘要 · Abstract (English)
Personalized alignment is crucial for enabling Large Language Models (LLMs) to engage effectively in user-centric interactions. However, current methods face a dual challenge: they fail to infer users' deep implicit preferences (including unstated goals, semantic context and risk tolerances), and they lack the defensive reasoning required to navigate real-world ambiguity. This cognitive gap leads to responses that are superficial, brittle and short-sighted. To address this, we propose Critique-Driven Reasoning Alignment (CDRA), which reframes alignment from a scalar reward-matching task into a structured reasoning process. First, to bridge the preference inference gap, we introduce the DeepPref benchmark. This dataset, comprising 3000 preference-query pairs across 20 topics, is curated by simulating a multi-faceted cognitive council that produces critique-annotated reasoning chains to deconstruct query semantics and reveal latent risks. Second, to instill defensive reasoning, we introduce the Personalized Generative Process Reward Model (Pers-GenPRM), which frames reward modeling as a personalized reasoning task. It generates a critique chain to evaluate a response's alignment with user preferences before outputting a final score based on this rationale. Ultimately, this interpretable, structured reward signal guides policy model through Critique-Driven Policy Alignment, a process-level online reinforcement learning algorithm integrating both numerical and natural language feedback. Experiments demonstrate that CDRA excels at discovering and aligning with users' true preferences while executing robust reasoning. Our code and dataset are available at https://github.com/Zephyrian-Hugh/Deep-pref.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。