TESSERA v2 提出高效训练遥感图像基础模型的实证方法，显著提升性能并降低部署成本。
TESSERA v2: Scaling Pixel-wise Earth Foundation Models
- 基于395次实验构建可控缩放研究，发现损失值无法预测下游效果
- 提出编码器与数据同步增长、投影层保持不变的算力分配规则
- 通过蒸馏生成小型模型，16维嵌入保留92%性能，存储仅需1/8
像素级地球观测（EO）基础模型正通过生成空间嵌入达到顶尖性能。然而，这些模型如何扩展以及如何最优利用预训练预算仍不清晰。我们开展了迄今最大的受控缩放研究：在固定像素级Barlow Twins框架下进行395次训练，每项在15个多样化下游任务上评估。结果表明，预训练损失几乎无法预测下游表现（|Pearson r| < 0.2），按损失选模型会浪费大量算力。我们还发现，随着训练预算增加，编码器和数据应协同增长，而投影层保持固定，由此得出简单的算力分配规则。据此训练了0.5B、1B和2B参数的像素级教师模型，并将最大模型蒸馏为紧凑学生模型用于嵌入即数据部署。总体而言，我们4400万参数的蒸馏学生模型优于所有测试的开源与专有嵌入产品，部分领先一个数量级。这些学生模型生成马特里什卡（Matryoshka）表示，16维前缀即可保留92%的128维完整性能，存储仅为1/8。综合结果提供了一套可落地的像素级EO基础模型扩展方案：训练大编码器、依据下游性能选择、蒸馏为灵活学生模型。我们计划发布TESSERA v2全球10米年均嵌入数据集，覆盖2017-2025年。全部代码已公开于：https://github.com/ucam-eo/tessera
原文摘要 · Abstract (English)
Pixel-wise Earth-observation (EO) foundation models are now achieving state-of-the-art performance via generated spatial embeddings. However, how these models scale and how best to spend a pretraining budget remain poorly understood. We present the largest controlled scaling study for EO to date: 395 training runs within a fixed pixel-wise Barlow Twins family, each evaluated on 15 diverse downstream tasks. We find that pretraining loss barely predicts downstream performance (|Pearson r| < 0.2), so selecting models by loss wastes a large share of the compute. We also find that, as the training budget grows, the encoder and the data should grow together while the projector stays fixed, which gives a simple rule for allocating compute. Using this rule, we train a family of pixel-wise teachers (0.5B, 1B, and 2B) and distil the largest into compact students for embeddings-as-data deployment. In aggregate, our 44-million-parameter distilled student outperforms every open and proprietary embedding product we test, several of them an order of magnitude larger. These students produce Matryoshka representations that are inexpensive to serve: a 16-dimensional prefix keeps 92% of the full 128-dimensional performance at 1/8 of the storage. Together, these results give a concrete, empirically grounded recipe for scaling pixel-wise EO foundation models: train large encoders, select by downstream performance, and distil into flexible student models. We plan to release global 10 m annual embeddings covering 2017-2025 as version 2 of the TESSERA foundation-model embeddings product. All code is available at: https://github.com/ucam-eo/tessera
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。