直接处理任意数量的图像集，兼顾特征提取与关系建模。
Convolutional Set Transformer
- 直接输入3D图像张量，边提取特征边建模图像间关系。
- 在图像集分类和异常检测任务上优于传统方法。
- 支持Grad-CAM等可视化解释，适合需要可解释性的场景。
我们提出卷积集合变换器（CST），一种新型神经架构，用于处理具有任意数量且视觉异质但共享高层语义（如类别、场景或概念）的图像集合。现有集合输入网络（如Deep Sets和Set Transformer）仅能处理向量输入，无法直接处理3D图像张量，需先通过CNN提取嵌入再建模图像间关系。相比之下，CST直接作用于3D图像张量，同步完成特征提取与上下文建模，实现两者的协同增益。该设计在图像集分类和异常检测任务中表现更优，并天然兼容如Grad-CAM等CNN可解释性方法，而其他方法则缺乏透明性。最后，我们证明CST可在大规模数据集上预训练，并通过标准迁移学习适配新领域与任务。为支持研究，我们发布了基于ImageNet预训练的CST-15模型（https://github.com/chinefed/convolutional-set-transformer）。
原文摘要 · Abstract (English)
We introduce the Convolutional Set Transformer (CST), a novel neural architecture designed to process image sets of arbitrary cardinality that are visually heterogeneous yet share high-level semantics - such as a common category, scene, or concept. Existing set-input networks, e.g., Deep Sets and Set Transformer, are limited to vector inputs and cannot directly handle 3D image tensors. As a result, they must be cascaded with a feature extractor, typically a CNN, which encodes images into embeddings before the set-input network can model inter-image relationships. In contrast, CST operates directly on 3D image tensors, performing feature extraction and contextual modeling simultaneously, thereby enabling synergies between the two processes. This design yields superior performance in tasks such as Set Classification and Set Anomaly Detection and further provides native compatibility with CNN explainability methods such as Grad-CAM, unlike competing approaches that remain opaque. Finally, we show that CSTs can be pre-trained on large-scale datasets and subsequently adapted to new domains and tasks through standard Transfer Learning schemes. To support further research, we release CST-15, a CST backbone pre-trained on ImageNet (https://github.com/chinefed/convolutional-set-transformer).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。