用自回归建模实现可控图像风格迁移,保持内容结构同时精准传递纹理。
StyleVAR: Controllable Image Style Transfer via Visual Autoregressive Modeling

- 将风格迁移建模为潜在空间中的条件离散序列生成,通过Transformer自回归预测目标图像
- 在多尺度上融合内容与风格信息,使用可调权重平衡二者影响,避免破坏生成连续性
- 两阶段训练提升性能,尤其在感知相似度指标上显著优于基线,适合风景与建筑图像
我们基于视觉自回归建模(VAR)框架,将风格迁移建模为学习潜在空间中的条件离散序列生成。图像通过VQ-VAE分解为多尺度表示并离散化为代码,再由Transformer自回归地建模目标代码分布,条件依赖于内容与风格代码。引入一种混合交叉注意力机制:目标表示自身历史作为键值,内容与风格特征作为查询,决定应强调历史中哪些部分。各尺度的融合系数控制风格与内容的相对影响,确保合成表示同时符合内容结构和风格纹理,且不破坏VAR的自回归连贯性。训练分两阶段进行:先在大规模三元组数据集上进行监督微调,再通过基于DreamSim的感知奖励,采用组相对策略优化(GRPO)进行强化学习微调,并使用动作归一化加权以均衡多尺度层级间的信用分配。在涵盖分布内、近分布及分布外三种场景的三个基准测试中,StyleVAR在风格损失、内容损失、LPIPS、SSIM、DreamSim和CLIP相似度上均持续优于AdaIN基线;强化学习阶段进一步提升了表现,尤其在奖励对齐的感知指标上。定性结果显示,该方法能有效转移纹理并保持语义结构,尤其适用于景观与建筑类图像;而对网络图像的泛化能力不足以及人脸处理困难,表明需增强内容多样性与更强的结构先验。
原文摘要 · Abstract (English)
We build on the Visual Autoregressive Modeling (VAR) framework and formulate style transfer as conditional discrete sequence modeling in a learned latent space. Images are decomposed into multi-scale representations and tokenized into discrete codes by a VQ-VAE; a transformer then autoregressively models the distribution of target tokens conditioned on style and content tokens. To inject style and content information, we introduce a blended cross-attention mechanism in which the evolving target representation attends to its own history, while style and content features act as queries that decide which aspects of this history to emphasize. A scale-dependent blending coefficient controls the relative influence of style and content at each stage, encouraging the synthesized representation to align with both the content structure and the style texture without breaking the autoregressive continuity of VAR. We train StyleVAR in two stages from a pretrained VAR checkpoint: supervised fine-tuning on a large triplet dataset of content--style--target images, followed by reinforcement fine-tuning with Group Relative Policy Optimization (GRPO) against a DreamSim-based perceptual reward, with per-action normalization weighting to rebalance credit across VAR's multi-scale hierarchy. Across three benchmarks spanning in-, near-, and out-of-distribution regimes, StyleVAR consistently outperforms an AdaIN baseline on Style Loss, Content Loss, LPIPS, SSIM, DreamSim, and CLIP similarity, and the GRPO stage yields further gains over the SFT checkpoint, most notably on the reward-aligned perceptual metrics. Qualitatively, the method transfers texture while maintaining semantic structure, especially for landscapes and architectural scenes, while a generalization gap on internet images and difficulty with human faces highlight the need for better content diversity and stronger structural priors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。