通过测试时采样优化大模型个性化，提升生成质量稳定性。
Test-Time Personalization: A Diagnostic Framework and Probabilistic Fix for Scaling Failures
- 测试时采样多个候选输出并选最优，实现推理阶段的可扩展个性化。
- 发现标准奖励模型存在用户级崩溃和查询级欺骗问题，导致性能停滞。
- 提出概率化奖励模型，有效缓解两类失败模式，提升多任务表现。
现有大模型个性化方法主要关注构建更好的个性化模型或输入，将推理视为单次过程。本文研究测试时个性化（TTP），通过从个性化策略模型中采样N个候选输出，并用个性化奖励模型选出最优解，实现推理阶段计算量的可扩展性。理论上证明，理想选择下期望效用随采样数对数增长，构成测试时扩展的理论上限。然而，标准奖励模型无法达到此上限。为诊断原因，我们推导出统一缩放定律，将任意奖励模型的Best-of-N曲线分解为四个可测量量，揭示两种失效模式：用户级坍塌（部分用户预测趋同）和查询级奖励欺骗（某些查询下预测与真实质量负相关）。基于该定律，我们提出一种概率化个性化奖励模型，其学习到的方差能有效缓解上述两种失效模式。实验验证了框架有效性：TTP在多个策略模型和个性化文本生成任务中均实现稳定扩展，且缩放定律与不同奖励模型变体的实际缩放曲线高度吻合。
原文摘要 · Abstract (English)
Existing approaches to LLM personalization focus on constructing better personalized models or inputs, while treating inference as a single-shot process. In this work, we study Test-Time Personalization (TTP) along an unexplored axis: scaling inference-time computation by sampling N candidates from a personalized policy model and selecting the best with a personalized reward model. We prove that oracle selection yields expected utility growing logarithmically with the number of sampled candidates, establishing a theoretical ceiling for test-time scaling. However, standard reward models fail to realize this potential. To diagnose why, we derive a unified scaling law that decomposes any reward model's Best-of-N curve into four measurable quantities and reveals two failure modes, user-level collapse (near-constant prediction for some users) and query-level reward hacking (negative correlation with true quality for some queries). Guided by this law, we propose a probabilistic personalized reward model whose learned variance effectively mitigates both failure modes. Experiments confirm both elements of our framework: TTP delivers consistent scaling across multiple policy models and personalized text generation tasks, and our scaling law closely matches observed scaling curves across reward-model variants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。