用视觉文本信息提升射电望远镜成像质量,减少伪影。
VVTRec: Radio Interferometric Reconstruction through Visual and Textual Modality Enrichment
- 将稀疏观测数据转为图像和文本特征,融合多模态信息增强重建
- 在不增加计算开销前提下,显著提升图像结构完整性和准确性
- 无需训练即可利用预训练视觉语言模型,适合天文图像处理场景
射电天文学依赖射电望远镜的波信号测量(称为可见性)进行远距离天体观测。这些可见性数据需转化为图像以供分析,但原始图像常混杂真实源信息与伪影。现有方法仅依赖单一模态的稀疏可见性数据,导致图像仍存伪影且相关性建模不足。为此,我们提出VVTRec,一种基于可见性引导的视觉与文本模态增强的多模态射电干涉数据重建方法。该方法将稀疏可见性转换为图像与文本特征,分别补充空间与语义信息,提升图像的结构完整性与精度。同时,利用预训练的视觉语言模型(VLMs)实现无须训练的性能提升,使未见过的稀疏可见性数据也能准确提取预训练知识作为补充。实验表明,VVTRec在不引入显著计算开销的情况下,有效利用多模态信息改善成像效果。
原文摘要 · Abstract (English)
Radio astronomy is an indispensable discipline for observing distant celestial objects. Measurements of wave signals from radio telescopes, called visibility, need to be transformed into images for astronomical observations. These dirty images blend information from real sources and artifacts. Therefore, astronomers usually perform reconstruction before imaging to obtain cleaner images. Existing methods consider only a single modality of sparse visibility data, resulting in images with remaining artifacts and insufficient modeling of correlation. To enhance the extraction of visibility information and emphasize output quality in the image domain, we propose VVTRec, a multimodal radio interferometric data reconstruction method with visibility-guided visual and textual modality enrichment. In our VVTRec, sparse visibility is transformed into image-form and text-form features to obtain enhancements in terms of spatial and semantic information, improving the structural integrity and accuracy of images. Also, we leverage Vision-Language Models (VLMs) to achieve additional training-free performance improvements. VVTRec enables sparse visibility, as a foreign modality unseen by VLMs, to accurately extract pre-trained knowledge as a supplement. Our experiments demonstrate that VVTRec effectively enhances imaging results by exploiting multimodal information without introducing excessive computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。