图像自然度能预测迁移效果,但并非原因,关键在局部结构。
Naturalness Predicts but Does Not Cause Transferability in Image Encodings of Real-World Streams
- 用频谱指数调节图像自然度,测试其对迁移性能的影响
- 自然度高时准确率不升反降,与结构信息无关
- 局部结构才是决定迁移能力的关键,适合研究模型泛化机制的人
将一维信号转为图像以复用预训练视觉模型是常见做法,但编码图像常被忽视。我们构建了WorldStream数据集,包含299个来自公开API的实时序列(天气、空气质量、地震、黄金原油、股票、加密货币、外汇、网络活动、空间天气),并设计九分类源识别任务,在3143个时间窗口上评估。在七种编码方式和六种冻结主干网络下,编码图像与自然图像的Frechet距离(FID)与迁移准确率呈强负相关(Spearman ρ = -0.72)。两个控制实验显示该关系非因果:调整频谱指数β使图像趋近自然分布,但准确率仅19.2%,远低于结构化基线的73.0%;而固定功率谱、打乱相位后,FID与准确率同步下降(Pearson r = -0.89)。结果表明,FID预测准确率是因为Inception模型捕捉到与主干网络相同的局部结构,而非自然度本身。全微调也未弥合差距(27% vs. 67%),说明缺陷源于结构本质。编码器可逆,从8位图像恢复信号信噪比达72.9 dB,图像可作为数据的无损记录。
原文摘要 · Abstract (English)
A common practice converts a one-dimensional signal into an image so that a vision backbone pretrained on natural photographs can be reused for recognition, yet the encoded image is rarely examined. We ask how the visual naturalness of an encoded image relates to its transfer accuracy under a frozen backbone. We build WorldStream, a corpus of 299 heterogeneous current-value series from key-free public APIs (weather, air quality, earthquakes, gold and oil, equities, crypto, foreign exchange, web activity and space weather), with a nine-way source-recognition task over 3143 temporally split windows. Across seven encodings and six frozen backbones, the Frechet distance of an encoding to natural images (FID) predicts its accuracy: Spearman $ρ=-0.72$. Two controlled interventions show this is not causal in the spectrum. Our invertible encoder has a single adjustable part, a spectral exponent $β$ (power $\propto |f|^{-β}$); varying $β$ moves the image toward or away from the natural-image manifold at fixed content. FID is lowest near the natural value $β\approx 2$, but frozen accuracy stays flat and far below the structured baselines (19.2% vs. 73.0%), and FID and accuracy are only weakly related over the sweep (Pearson $-0.32$). A second intervention, phase scrambling, holds the power spectrum exactly fixed while removing local structure; now FID and accuracy fall together (Pearson $-0.89$). The cross-encoding correlation is thus mediated by local structure, not spectral naturalness: FID predicts accuracy because Inception reads the same structure the backbones do. Full fine-tuning does not close the gap (27% vs. 67%), so the deficit is structural. The encoder is exactly invertible, recovering the signal from the 8-bit image at 72.9 dB, so the image doubles as a lossless record of the data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。