arXiv:2605.08254cs.LGcs.AI2026-05

让文生图模型快速、稳定地控制生成内容,支持任意新概念即时生效。

HyperTransport: Amortized Conditioning of T2I Generative Models

论文配图:HyperTransport: Amortized Conditioning of T2I Generative Models
图 1 · 摘自论文原文
  • 用超网络将文本编码直接映射为调控参数,避免每次调参都要重算。
  • 新概念生成速度提升3600至7000倍,效果媲美专门训练的基线方法。
  • 支持图像直接引导文字生成,适合需要灵活控制的创意应用。

随着基础模型能力增强,高效可靠地控制其行为变得至关重要。微调成本高昂,而提示词控制虽便捷但对措辞敏感,易失效。现有激活调控方法需针对每个概念单独优化,导致在概念集合大、动态变化或按需指定时效率极低,每次新增概念至少需数分钟优化。本文提出HyperTransport,一种超网络框架,通过预训练编码器(如CLIP)的嵌入直接映射干预参数,端到端使用最优传输损失训练。训练后,每项新干预仅需一次超网络前向传播,速度比逐概念拟合快3600–7000倍。在未见概念上,其表现可媲美最强的逐概念基线。通过解耦概念表示与干预预测,HyperTransport首次实现三项能力的统一:面向开放概念集的代价分摊调控、连续可解释的强度控制、以及参考图像直接驱动文本生成的跨模态条件生成。我们在DMD2和Nitro-1-PixArt上对167个保留测试概念进行了验证,采用基于CLIP的指标、视觉语言模型作为评判者及用户研究。成对对比中,人类与VLM评委偏好HyperTransport的频率是提示词的两倍以上。

原文摘要 · Abstract (English)

As foundation models grow in capability, the ability to efficiently and reliably control their behavior becomes critical. Fine-tuning these models can be costly, and while prompting can be practical for controllability, it remains fragile due to models' high sensitivity to exact prompt wording and structure. This brittleness has driven interest in activation steering techniques that offer more stable and predictable control over model behavior. However, existing activation steering methods require per-concept optimization, which makes them ill-suited to deployment scenarios where the concept set is large, evolving, or only specified at request time: each new concept incurs at least minutes of optimization on the target model. We propose HyperTransport, a hypernetwork framework that amortizes this cost by mapping embeddings from a pretrained encoder (CLIP in our instantiation) directly to intervention parameters, trained end-to-end using an optimal transport loss. Once trained, HyperTransport produces each new intervention in a single hypernetwork forward pass, 3600-7000x faster than per-concept fitting. On concepts unseen during training, it matches the strongest per-concept baselines at inducing the target concept. By decoupling concept representation from intervention prediction, HyperTransport combines three capabilities that no existing approach offers as a set: amortized steering for open-ended concept sets, continuous interpretable strength control, and cross-modal conditioning where reference images can directly steer text-based generation. We validate HyperTransport on DMD2 and Nitro-1-PixArt across 167 held-out test concepts via CLIP-based metrics, a VLM-as-a-judge evaluation, and a user study. In pairwise comparisons, both human and VLM judges prefer HyperTransport over prompting ~2x as often.

文生图可控生成超网络跨模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。