首次测试先进表格模型作图像-表格融合的编码器，发现标签依赖成关键瓶颈。
The Importance of Encoder Choice:A Tabular-Image Study
- 用最新表格模型替代传统MLP做表格编码，探索多模态学习新路径
- 基于上下文学习的模型需标签才能处理样本，训练与测试无法统一编码
- 揭示编码器选择对多模态学习至关重要，尤其在表格数据场景
多模态学习通常为每种模态配置专用编码器。当涉及表格模态时，以往研究大多使用简单的前馈神经网络（plain MLP）作为编码器。然而，若表格领域已有强大编码器，它便不会被称为深度学习最后未征服的堡垒。本研究首次在图像-表格设定中评估最先进的表格模型作为编码器的表现。一个显著障碍浮现：在表格领域表现优异的上下文学习（In-Context Learning）模型需要标签来处理实例，导致训练和测试样本难以采用一致方式嵌入。我们针对该家族多个模型解决了这一问题。本研究旨在强调编码器选择在多模态学习中的重要性。
原文摘要 · Abstract (English)
Multimodal learning usually requires a dedicated encoder per modality. When a tabular modality is involved, prior work has been mostly using a \emph{plain MLP} as the encoder. Yet if it were a strong encoder, the tabular domain would not be ``the last unconquered castle for deep learning''. This study evaluates state-of-the-art tabular models as encoders in the image-tabular setting for the first time. An obstacle stands out. In-Context Learning models, among the best performing methods in the tabular domain, require labels to process instances, making it non-trivial to embed training and test instances the same way. We addressed this problem across multiple models of this family. With this study, we would like to highlight the importance of encoder factor in the multimodal learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。