用低成本辅助信息加速高成本显微成像，提升重建质量
Leveraging Multimodal Diffusion Models to Accelerate Imaging with Side Information
- 构建多模态扩散模型，将复杂逆问题转化为线性补全
- 仅需少量昂贵显微数据即可实现高质量图像重建
- 适合材料科学等需要多模态成像的科研人员
扩散模型在解决逆问题中表现出色，但其在结构化科学领域的应用仍受限。针对材料科学中的成像需求，我们提出利用廉价辅助模态的侧信息，减少对高成本成像模态的测量次数。为应对前向模型不可导且黑箱的问题，我们设计框架在联合模态上训练多模态扩散模型，将黑箱前向模型的逆问题转化为简单的线性插值问题。数值实验表明，该方法可在材料影像数据上成功训练扩散模型，显著降低对昂贵显微数据的需求，同时实现更优的图像重建效果。
原文摘要 · Abstract (English)
Diffusion models have found phenomenal success as expressive priors for solving inverse problems, but their extension beyond natural images to more structured scientific domains remains limited. Motivated by applications in materials science, we aim to reduce the number of measurements required from an expensive imaging modality of interest, by leveraging side information from an auxiliary modality that is much cheaper to obtain. To deal with the non-differentiable and black-box nature of the forward model, we propose a framework to train a multimodal diffusion model over the joint modalities, turning inverse problems with black-box forward models into simple linear inpainting problems. Numerically, we demonstrate the feasibility of training diffusion models over materials imagery data, and show that our approach achieves superior image reconstruction by leveraging the available side information, requiring significantly less amount of data from the expensive microscopy modality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。