用单调性约束让模型学出更小更稳的高保真表示,性能几乎不降。
MonoCon: A general framework for learning ultra-compact high-fidelity representations using monotonicity constraints
- 在预训练编码器后加小单调MLP头,通过对比损失和单调约束共同优化
- CIFAR-100上表示压缩近9倍,鲁棒性提升1.5倍,准确率保留99%
- 适合边缘计算、大规模检索等对紧凑性和稳定性要求高的场景
学习高质量、鲁棒、高效且解耦的表征是人工智能的核心挑战。现有深度度量学习框架主要依赖架构与优化约束。本文提出第三种路径:功能约束。我们介绍MonoCon,一种简单框架,在任意预训练编码器后接一个小型单调多层感知机(MLP)头。在对比损失和单调性约束引导下,编码器与头协同适应,使MonoCon学习到鲁棒、解耦且极紧凑的嵌入,性能损耗可忽略。在CIFAR-100图像分类任务中,相比微调编码器基线,表示压缩近9倍,鲁棒性提升1.5倍,同时保持99%的5-NN分类准确率。在SNLI句子相似性任务中,表示压缩3.4倍,鲁棒性提升1.4倍,仅小幅降低STSb得分。证明了MonoCon作为通用无领域依赖框架的潜力。这些通过功能约束获得的鲁棒超紧凑表示,为边缘计算到云规模检索等不同场景提供了统一解决方案。
原文摘要 · Abstract (English)
Learning high-quality, robust, efficient, and disentangled representations is a central challenge in artificial intelligence (AI). Deep metric learning frameworks tackle this challenge primarily using architectural and optimization constraints. Here, we introduce a third approach that instead relies on $\textit{functional}$ constraints. Specifically, we present MonoCon, a simple framework that uses a small monotonic multi-layer perceptron (MLP) head attached to any pre-trained encoder. Due to co-adaptation between encoder and head guided by contrastive loss and monotonicity constraints, MonoCon learns robust, disentangled, and highly compact embeddings at a practically negligible performance cost. On the CIFAR-100 image classification task, MonoCon yields representations that are nearly 9x more compact and 1.5x more robust than the fine-tuned encoder baseline, while retaining 99\% of the baseline's 5-NN classification accuracy. We also report a 3.4x more compact and 1.4x more robust representation on an SNLI sentence similarity task for a marginal reduction in the STSb score, establishing MonoCon as a general domain-agnostic framework. Crucially, these robust, ultra-compact representations learned via functional constraints offer a unified solution to critical challenges in disparate contexts ranging from edge computing to cloud-scale retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。