通过强化学习统一个性化理解与生成,提升推理能力。
Uni-Synergy: Bridging Understanding and Generation for Personalized Reasoning via Co-operative Reinforcement Learning

- 设计联合优化框架Sync-R1,在同一推理循环中同步训练理解与生成。
- 在UnifyBench++上实现顶尖性能,跨任务推理与个性化表现优异。
- 适合需要真实用户上下文的对话系统、内容生成研究者使用。
统一多模态模型(UMMs)在通用任务中表现优秀,但在个性化理解与生成之间仍存在鸿沟。以往方法主要依赖监督微调进行隐式标记级对齐,难以充分捕捉理解与创作间的协同效应。本文提出Sync-R1,一个端到端强化学习框架,将个性化理解与生成纳入单一显式推理循环中联合优化。通过统一反馈机制,个性化理解引导内容生成,生成质量又反哺理解优化,形成集成奖励空间。为高效实现双任务协同,提出Sync-GRPO,采用集成奖励系统;并引入动态分组缩放(DGS),自适应过滤低潜力轨迹,降低梯度方差,加速收敛。为更贴近现实复杂性,构建了包含更密集文本描述与丰富用户上下文的UnifyBench++基准。实验表明,Sync-R1达到当前最优性能,展现卓越的跨任务推理与鲁棒个性化能力,且无需复杂冷启动流程。代码与数据集将开源于:https://github.com/arctanxarc/UniCTokens。
原文摘要 · Abstract (English)
Unified Multimodal Models (UMMs) excel in general tasks but struggle to bridge the gap between personalized understanding and generation. Prior works largely rely on implicit token-level alignment via supervised fine-tuning, which fails to fully capture the potential synergy between comprehension and creation. In this work, we propose Sync-R1, an end-to-end reinforcement learning framework that jointly optimizes personalized understanding and generation within a single, explicit reasoning loop. Through this unified feedback process, Sync-R1 enables personalized comprehension to guide content creation, while the resulting generation quality reciprocally refines understanding within an integrated reward landscape. To efficiently orchestrate this dual-task synergy, we introduce Sync-GRPO, a reinforcement learning method utilizing an ensemble reward system. Furthermore, we propose Dynamic Group Scaling (DGS), which adaptively filters low-potential trajectories to reduce gradient variance and accelerate convergence. To better reflect real-world complexity, we introduce UnifyBench++, featuring denser textual descriptions and richer user contexts. Experimental results demonstrate that Sync-R1 achieves state-of-the-art performance, showcasing superior cross-task reasoning and robust personalization without requiring complex cold-start procedures. The code and the UnifyBench++ dataset will be released at: https://github.com/arctanxarc/UniCTokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。