用多模态大模型提升小样本学习,让科学领域少数据也能高效建模。
A Foundational Multi-Modal Model for Few-Shot Learning
- 基于跨域多模态数据训练大模型,提升小样本泛化能力
- 在10000+样本的M3FD数据集上实现超越传统元学习的效果
- 专为科研场景设计,支持医学、材料等数据稀缺领域使用
小样本学习(FSL)旨在仅用少量标注样本(每类通常少于10个)就实现模型泛化,对生物医学、环境、材料和机械科学尤为重要,因这些领域样本稀缺且采集成本高。本文提出一种创新方法:通过在涵盖多种领域、任务类型和输入模态的独立任务上训练大型多模态模型(LMMM),显著提升FSL模型的泛化性能,优于传统元学习模型。为此,我们构建了包含超过10,000个样本的多模态小样本数据集M3FD,涵盖2D RGB图像、2D/3D医学影像、表格与时间序列数据,并人工标注了分类等小样本任务。进一步提出了M3F框架——专为数据受限科学应用设计的大型多模态模型架构,通过模块化流水线支持广泛科学数据类型。在M3FD上微调后,M3F显著提升模型性能,使LMMM在真实小样本场景中部署成为可能。源码位于https://github.com/ptdang1001/M3F。为促进可复现性与开放使用,M3FD配套提供灵活易用工具,支持高效查询、任务特异性采样与预处理。整体方案提供统一、可扩展的解决方案,大幅降低科学领域应用多模态大模型的门槛。
原文摘要 · Abstract (English)
Few-shot learning (FSL) is a machine learning paradigm that aims to generalize models from a small number of labeled examples, typically fewer than 10 per class. FSL is particularly crucial in biomedical, environmental, materials, and mechanical sciences, where samples are limited and data collection is often prohibitively costly, time-consuming, or ethically constrained. In this study, we present an innovative approach to FSL by demonstrating that a Large Multi-Modal Model (LMMM), trained on a set of independent tasks spanning diverse domains, task types, and input modalities, can substantially improve the generalization of FSL models, outperforming models based on conventional meta-learning on tasks of the same type. To support this, we first constructed a Multi-Modal Model Few-shot Dataset (M3FD, over 10K+ few-shot samples), which includes 2D RGB images, 2D/3D medical scans, tabular and time-course datasets, from which we manually curated FSL tasks such as classification. We further introduced M3F (Multi-Modal Model for Few-shot learning framework), a novel Large Multi-Modal Model framework tailored for data-constrained scientific applications. M3F supports a wide range of scientific data types through a modular pipeline. By fine-tuning the model on M3FD, M3F improves model performance, making LMMM feasible for real-world FSL deployment. The source code is located at https://github.com/ptdang1001/M3F. To democratize access to complex FSL data and promote reproducibility for public usage, M3FD is paired with a flexible and user-friendly tool that enables efficient querying, task-specific sampling, and preprocessing. Together, our dataset and framework offer a unified, scalable solution that significantly lowers the barrier to applying LMMMs in data-scarce scientific domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。