让个性化图像生成更快更准,通过智能选模型+精准量化
PersonalQ: Select, Quantize, and Serve Personalized Diffusion Models for Efficient Inference

- 用触发词统一管理模型选择与量化,提升准确率
- 仅在关键路径保留高精度,压缩比超传统方法
- 适合需要高效部署大量个人化模型的开发者
个性化文本到图像生成允许用户将扩散模型微调为特定概念的检查点仓库,但高效服务这些仓库面临两大挑战:自然语言请求常含歧义,易被误导向视觉相似的检查点;标准后训练量化会破坏编码个性化概念的脆弱表征。我们提出 PersonalQ,一个通过共享触发词信号连接检查点选择与量化的统一框架。Check-in 通过意图感知混合检索与基于大语言模型的重排序,实现对齐意图的选择,并在存在多个可能意图时仅提出简短澄清问题;随后插入所选检查点的规范触发词重写提示。与此同时,触发词感知量化(TAQ)在交叉注意力中应用触发词感知的混合精度,保留触发词条件的关键/值行及其注意力权重,同时对其他路径进行激进量化,实现内存高效推理。实验表明,PersonalQ 在意图对齐上优于检索与重排序基线,而 TAQ 始终在压缩率-质量权衡上超越先前的扩散模型后训练量化方法,使个性化检查点的可扩展服务成为可能,且不牺牲保真度。
原文摘要 · Abstract (English)
Personalized text-to-image generation lets users fine-tune diffusion models into repositories of concept-specific checkpoints, but serving these repositories efficiently is difficult for two reasons: natural-language requests are often ambiguous and can be misrouted to visually similar checkpoints, and standard post-training quantization can distort the fragile representations that encode personalized concepts. We present PersonalQ, a unified framework that connects checkpoint selection and quantization through a shared signal -- the checkpoint's trigger token. Check-in performs intent-aligned selection by combining intent-aware hybrid retrieval with LLM-based reranking over checkpoint context and asks a brief clarification question only when multiple intents remain plausible; it then rewrites the prompt by inserting the selected checkpoint's canonical trigger. Complementing this, Trigger-Aware Quantization (TAQ) applies trigger-aware mixed precision in cross-attention, preserving trigger-conditioned key/value rows (and their attention weights) while aggressively quantizing the remaining pathways for memory-efficient inference. Experiments show that PersonalQ improves intent alignment over retrieval and reranking baselines, while TAQ consistently offers a stronger compression-quality trade-off than prior diffusion PTQ methods, enabling scalable serving of personalized checkpoints without sacrificing fidelity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。