用CLIP对齐图文，从网页自动摘要生成多模态内容。
Web-Scale Multimodal Summarization using CLIP-Based Semantic Alignment
- 通过CLIP模型衡量图像与文本的语义匹配度，实现跨模态排序。
- 在500组图文对上达到96.99%准确率和0.9270的ROC-AUC。
- 支持可调参数与自定义配置，适合研究者快速部署使用。
我们提出Web-Scale Multimodal Summarization，一个轻量级框架，通过整合从网络检索的文本与图像数据生成摘要。用户输入主题后，系统并行执行网页、新闻和图像搜索。利用微调后的CLIP模型对检索到的图像进行语义对齐排序，以匹配主题与文本。可选的BLIP图像描述功能支持仅基于图像生成摘要，增强多模态一致性。该流程支持可调节的获取上限、语义过滤、摘要样式设置及结构化输出下载。系统通过Gradio API对外提供服务，支持参数可控与预设配置。在包含500组图像-标题对、20:1对比负样本的评估中，取得0.9270的ROC-AUC、0.6504的F1分数和96.99%的准确率，验证了强大的多模态对齐能力。本工作提供了一个可配置、可部署的工具，将语言、检索与视觉模型集成于用户可扩展的流水线中。
原文摘要 · Abstract (English)
We introduce Web-Scale Multimodal Summarization, a lightweight framework for generating summaries by combining retrieved text and image data from web sources. Given a user-defined topic, the system performs parallel web, news, and image searches. Retrieved images are ranked using a fine-tuned CLIP model to measure semantic alignment with topic and text. Optional BLIP captioning enables image-only summaries for stronger multimodal coherence.The pipeline supports features such as adjustable fetch limits, semantic filtering, summary styling, and downloading structured outputs. We expose the system via a Gradio-based API with controllable parameters and preconfigured presets.Evaluation on 500 image-caption pairs with 20:1 contrastive negatives yields a ROC-AUC of 0.9270, an F1-score of 0.6504, and an accuracy of 96.99%, demonstrating strong multimodal alignment. This work provides a configurable, deployable tool for web-scale summarization that integrates language, retrieval, and vision models in a user-extensible pipeline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。