arXiv:2507.23372cs.CV2025-07中稿 · TIP 2026被引 16

统一情绪理解与生成,用可学习专家查询实现双向增强。

UniEmo: Unifying Emotional Understanding and Generation with Learnable Expert Queries

  • 构建分层专家查询链,提取多尺度情绪特征以统一任务
  • 融合情绪相关系数与条件损失,提升生成图像的多样性和真实性
  • 双向反馈机制让生成反哺理解,适合情绪生成与分析研究者

情绪理解与生成常被视作独立任务,但二者本质互补且可相互促进。本文提出UniEmo,一个统一框架,通过可学习专家查询构建分层情绪理解链,逐步提取多尺度情绪特征,为统一任务奠定基础。同时,将专家查询与情绪表征融合,引导扩散模型生成具情绪感染力的图像。为提升生成多样性与保真度,引入情绪相关系数与情绪条件损失,实现生成过程中的对齐与优化。实验表明,联合训练使生成模块为理解部分提供隐式反馈;进一步设计新型数据过滤算法,筛选高质量多样化生成图像,显式反馈至理解模块。双路径生成驱动反馈显著增强模型情绪理解能力。大量实验显示,UniEmo在情绪理解与生成任务上均显著优于现有方法。代码已开源:https://github.com/JiuTian-VL/UniEmo。

原文摘要 · Abstract (English)

Emotional understanding and generation are often treated as separate tasks, yet they are inherently complementary and can mutually enhance each other. In this paper, we propose the UniEmo, a unified framework that seamlessly integrates these two tasks. The key challenge lies in the abstract nature of emotions, necessitating the extraction of visual representations beneficial for both tasks. To address this, we propose a hierarchical emotional understanding chain with learnable expert queries that progressively extracts multi-scale emotional features, thereby serving as a foundational step for unification. Simultaneously, we fuse these expert queries and emotional representations to guide the diffusion model in generating emotion-evoking images. To enhance the diversity and fidelity of the generated emotional images, we further introduce the emotional correlation coefficient and emotional condition loss into the fusion process. This step facilitates fusion and alignment for emotional generation guided by the understanding. In turn, we demonstrate that joint training allows the generation component to provide implicit feedback to the understanding part. Furthermore, we propose a novel data filtering algorithm to select high-quality and diverse emotional images generated by the well-trained model, which explicitly feedback into the understanding part. Together, these generation-driven dual feedback processes enhance the model's understanding capacity. Extensive experiments show that UniEmo significantly outperforms state-of-the-art methods in both emotional understanding and generation tasks. The code for the proposed method is available at https://github.com/JiuTian-VL/UniEmo.

情绪生成扩散模型双向反馈多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。