针对模拟存内计算噪声，提出无需重训练的引导校准方法，恢复扩散模型生成质量。
When Guidance Goes Off-Scale: Recalibrating Diffusion Transformers under Analog Compute-in-Memory Nonidealities
- 通过分析噪声对引导残差的影响，识别出可调控的失真通道。
- 在不同噪声水平下动态调整引导尺度，使生成更贴近提示语义。
- 无需重训练即可修复87%以上因硬件非理想性导致的图像质量下降，适合边缘部署场景。
扩散Transformer（DiTs）因采样时反复调用大型去噪器而产生高内存流量与能耗，其核心为线性运算。模拟存内计算（CIM）可通过在存储权重的内存阵列中执行线性运算缓解此问题。然而，CIM非理想性会扰动有效权重，误差沿状态依赖的去噪轨迹累积；其与无分类器引导（CFG）的交互尚不明确。本文系统分析了模拟CIM非理想性对DiT采样的影响。尽管条件与无条件预测各自仍接近理想值，但它们的差值（即CFG残差）被显著衰减并旋转。我们识别该残差为可控失效通道，提出一种无需重训练、仅在采样端调节引导尺度的校准方法。轨迹级分析表明，适度校准可增强畸变残差中保留的目标导向成分，促使更早进入与提示一致的语义区域。相反，过度引导会放大完整噪声残差，降低生成质量，存在噪声依赖的最优解。在PixArt-Sigma、PixArt-alpha和DiT-XL/2上大量实验表明，最优引导尺度随CIM噪声增大而上升。使用每条件30,000样本测试，校准方法在模拟的CIM映射下持续恢复生成质量，至少关闭87%的CIM-induced FID差距（噪声水平0.20）。具体表现为：PixArt-Sigma的FID从59.22降至20.49，PixArt-alpha从72.37降至21.12，DiT-XL/2从20.89降至6.62。
原文摘要 · Abstract (English)
Diffusion Transformers (DiTs) incur high memory traffic and energy costs because sampling repeatedly evaluates large denoisers dominated by linear operations. Analog compute-in-memory (CIM) can alleviate these costs by executing linear operations within weight-storing memory arrays. However, CIM nonidealities perturb effective weights, with errors accumulating along the state-dependent denoising trajectory; their interaction with classifier-free guidance (CFG) remains underexplored. In this paper, we characterize the impact of analog CIM nonidealities on DiT sampling. Although conditional and unconditional predictions can each remain close to their clean counterparts, their difference (the CFG residual) is disproportionately attenuated and rotated. Identifying this residual as a controllable failure channel, we propose a retraining-free, sampler-side recalibration that adjusts only the CFG scale for a given CIM condition. Trajectory-level analysis shows that moderate recalibration strengthens the target-oriented component preserved in the distorted residual, enabling earlier commitment to a prompt-consistent semantic region. In contrast, excessive guidance amplifies the full noisy residual and degrades quality, resulting in a finite, noise-dependent optimum. Extensive experiments on PixArt-Sigma, PixArt-alpha, and DiT-XL/2 show that the optimal guidance scale increases with CIM noise. Using 30,000 samples per condition, guidance recalibration consistently restores generation quality across simulated CIM mappings, closing at least 87% of the CIM-induced FID gap at a CIM noise level of 0.20. It reduces FID from 59.22 to 20.49 on PixArt-Sigma, 72.37 to 21.12 on PixArt-alpha, and 20.89 to 6.62 on DiT-XL/2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。