用表格大模型解决近红外光谱校准难题,小样本下表现优于传统方法。
Tabular foundation models for robust calibration of near-infrared chemical sensing data

- 直接用表格大模型处理原始光谱数据,避免复杂预处理
- 在66个数据集上平均排名领先,回归任务显著优于偏最小二乘法
- 适合小中规模校准场景,但对异常值和外推样本仍需改进
近红外光谱作为一种快速无损的化学传感技术,广泛应用于食品、药品、生物和环境样品分析。然而其实际应用仍依赖于能处理高维共线光谱、样本量有限、预处理敏感、光谱异常值及域外外推等问题的校准模型。本文评估了表格基础模型在近红外化学传感校准中的可行性。在涵盖54个回归与12个分类任务的66个近红外数据集上,对比了TabPFN在原始光谱上的直接推理与优化预处理后的推理,分别与PLS/PLS-DA、Ridge、CatBoost及一维卷积神经网络进行比较。采用统一验证框架,预处理与模型选择仅基于校准数据,外部测试前完成。回归任务中,优化预处理的TabPFN取得最佳平均排名,显著优于原始光谱上的TabPFN、PLS、CNN-1D,与Ridge统计相当;分类任务中,直接使用原始光谱的TabPFN表现最佳,接近优化版本。鲁棒性分析显示,尽管平均预测性能强,但在光谱异常值和外推样本上优势减弱,经典化学位模型仍具竞争力。结果表明,表格基础模型可补充现有化学位模型流程,尤其适用于小至中等规模校准,但也凸显出对光谱特定先验知识与不确定性感知部署策略的需求。
原文摘要 · Abstract (English)
Near-infrared spectroscopy is increasingly used as a rapid, non-destructive chemical sensing technology for the analysis of food, pharmaceutical, biological, and environmental samples. However, the practical deployment of NIR sensors still depends on calibration models able to handle high-dimensional, collinear spectra, limited sample sizes, preprocessing dependence, spectral outliers, and extrapolation beyond the calibration domain. Here, we evaluate whether tabular foundation models can provide a new calibration strategy for NIR chemical sensing. We benchmark TabPFN on 66 NIR datasets covering 54 regression and 12 classification tasks, and compare direct inference on raw spectra with preprocessing-optimized inference against PLS/PLS-DA, Ridge, Catboost, and one-dimensional convolutional neural networks. The study uses a unified validation framework in which preprocessing and model selection are performed exclusively on calibration data before external test evaluation. In regression, preprocessing-optimized TabPFN achieves the best overall average rank and significantly outperforms PLS, CatBoost, TabPFN on raw spectra, and CNN-1D, while remaining statistically comparable to Ridge. In classification, TabPFN applied directly to raw spectra provides the best average rank, with performance close to the optimized variant. Robustness analyses show that TabPFN provides strong average predictive performance but that its advantage decreases on spectral outliers and extrapolated samples, where classical chemometric models remain competitive. These results suggest that tabular foundation models can complement established chemometric workflows for NIR chemical sensing, especially in small- to medium-sized calibration settings, while highlighting the need for spectroscopy-specific priors and uncertainty-aware deployment strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。