让大模型学会推理用户偏好，生成更个性化长文本。
Reasoning-Enhanced Self-Training for Long-Form Personalized Text Generation
- 通过生成推理路径训练模型理解用户背景与风格。
- 在LongLaMP上平均性能提升14.5%，超越现有方法。
- 适合需要深度个性化输出的场景，如定制写作与对话。
个性化文本生成要求大语言模型具备从非标准训练数据中学习的能力。为使模型更好利用用户过往偏好、背景知识或写作风格，我们提出一种名为REST-PG的框架，通过在生成过程中推理个人数据来提升输出相关性。REST-PG首先生成推理路径以训练模型的推理能力，再采用期望最大化强化自训练机制，基于模型自身高奖励输出迭代优化。我们在包含四项多样化任务的LongLaMP基准上评估该方法，实验表明，REST-PG在该基准上相比当前最优基线实现平均14.5%的相对性能提升。
原文摘要 · Abstract (English)
Personalized text generation requires a unique ability of large language models (LLMs) to learn from context that they often do not encounter during their standard training. One way to encourage LLMs to better use personalized context for generating outputs that better align with the user's expectations is to instruct them to reason over the user's past preferences, background knowledge, or writing style. To achieve this, we propose Reasoning-Enhanced Self-Training for Personalized Text Generation (REST-PG), a framework that trains LLMs to reason over personal data during response generation. REST-PG first generates reasoning paths to train the LLM's reasoning abilities and then employs Expectation-Maximization Reinforced Self-Training to iteratively train the LLM based on its own high-reward outputs. We evaluate REST-PG on the LongLaMP benchmark, consisting of four diverse personalized long-form text generation tasks. Our experiments demonstrate that REST-PG achieves significant improvements over state-of-the-art baselines, with an average relative performance gain of 14.5% on the benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。