快速预测个性化生成模型适配参数，秒级完成微调。
LoFA: Learning to Predict Personalized Priors for Fast Adaptation of Visual Generative Models
- 通过两阶段超网络预测LoRA参数分布模式。
- 在多个任务上秒级生成高质量适配参数，超越传统LoRA。
- 适合需要快速个性化定制视觉生成模型的用户。
个性化视觉生成模型以满足特定用户需求日益受到关注，但现有方法如低秩适配（LoRA）因需任务专属数据且优化耗时而难以实用。少数基于超网络的方法虽可直接预测适配权重，却难以将细粒度用户提示映射到复杂的LoRA分布，限制了实用性。为此，我们提出LoFA框架，高效预测个性化先验以实现快速模型适配。首先，我们发现LoRA的一个关键特性：LoRA与基模型参数间的相对变化呈现结构化分布模式。基于此，设计两阶段超网络：先预测捕捉关键适配区域的相对分布模式，再据此引导最终LoRA权重生成。大量实验表明，该方法在多个任务和用户提示下均能在数秒内稳定预测高质量个性化先验，性能甚至优于需数小时处理的传统LoRA。项目页面：https://jaeger416.github.io/lofa/。
原文摘要 · Abstract (English)
Personalizing visual generative models to meet specific user needs has gained increasing attention, yet current methods like Low-Rank Adaptation (LoRA) remain impractical due to their demand for task-specific data and lengthy optimization. While a few hypernetwork-based approaches attempt to predict adaptation weights directly, they struggle to map fine-grained user prompts to complex LoRA distributions, limiting their practical applicability. To bridge this gap, we propose LoFA, a general framework that efficiently predicts personalized priors for fast model adaptation. We first identify a key property of LoRA: structured distribution patterns emerge in the relative changes between LoRA and base model parameters. Building on this, we design a two-stage hypernetwork: first predicting relative distribution patterns that capture key adaptation regions, then using these to guide final LoRA weight prediction. Extensive experiments demonstrate that our method consistently predicts high-quality personalized priors within seconds, across multiple tasks and user prompts, even outperforming conventional LoRA that requires hours of processing. Project page: https://jaeger416.github.io/lofa/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。