LLM在主观任务中易陷入推理崩溃,本文提出动态路由与长度约束策略提升准确率。
LLM Reasoning for Subjective Tasks: Failure Modes, Mitigation, and Dynamic Reasoning Routing

- 通过条件长度惩罚后训练,防止推理过长导致的性能崩溃
- 不同推理角色下验证准确率波动达0.38宏F1,证明风格适配关键
- 提出动态推理路由架构,适配真实场景中的主观评判需求
推荐系统依赖个性化,其正确性本质是主观偏好而非客观真理。当大语言模型(LLMs)被用作自主安全与质量验证器时,面临上下文感知的偏好对齐挑战。尽管强化学习结合可验证奖励(RLVR)在客观数学任务上取得进展,但其在真实生产环境中的四个主观验证任务上表现不佳。大规模研究发现:僵化的数学式推理会削弱验证效果,标准RLVR引发‘推理崩溃’——策略放弃深度思考转而快速启发式猜测。本文提出一种条件长度惩罚的后训练算法,将验证准确率与受限推理长度绑定,有效遏制崩溃并恢复性能。进一步发现,推理风格与社会语言框架密切相关:在1500个合成人格下,验证准确率(宏F1)变化高达0.38,表明多数主观验证错误实为推理风格不匹配所致。由此提出一种训练中期的推理路由架构,根据上下文动态选择适配的人格化推理路径。本工作提供可扩展的算法修补方案与长期的架构蓝图,推动推理模型与真实世界主观约束对齐。
原文摘要 · Abstract (English)
Recommendation systems thrive on personalization, where ''correctness'' is rarely a binary truth but a matter of subjective human preference. As Large Language Models (LLMs) are deployed as autonomous verifiers of safety and quality guidelines, they face a distinctive challenge: context-aware preference alignment. Recent gains in Reinforcement Learning with Verifiable Rewards (RLVR) are indexed mostly on objective, mathematical tasks. Through a large-scale study spanning both proprietary and open-source models on four real-world verification tasks from a production recommender platform, we ask whether explicit reasoning generalizes to subjective, human-centric industry rubrics. We expose a fundamental vulnerability: rigid, math-centric reasoning traces actively degrade verification, and applying standard RLVR triggers a phenomenon we term reasoning collapse, in which the policy abandons deliberation in favor of rapid heuristic guessing. We introduce a conditional length-penalized post-training algorithm that intertwines verification accuracy with bounded reasoning length, halting collapse and recovering performance. Finally, we show that a reasoning trace's efficacy is tightly coupled with its socio-linguistic framing: across 1500 synthesized personas, verification accuracy swings by nearly 0.38 macro-F1 depending solely on the adopted reasoning persona---evidence that much subjective-verification error is really reasoning-style mismatch. This observation motivates a mid-training architecture that routes reasoning through contextually aligned personas. This work offers both a scalable algorithmic patch and a long-term architectural blueprint for aligning reasoning models with real-world subjective constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。