arXiv:2504.18562cs.LGcs.AI2025-04被引 1

用Gemma 3的内部知识层预测野火,提升准确率且节省数据。

Deep Learning with Pretrained 'Internal World' Layers: A Gemma 3-Based Modular Architecture for Wildfire Prediction

  • 用Gemma 3中间层作内部世界,冻结其参数只训练输入输出模块。
  • 在摩洛哥数据集上预测准确率优于传统模型,鲁棒性更强。
  • 适合资源有限但需高可靠性的环境风险预测场景。

深度学习模型,尤其是大型Transformer,其中间层蕴含大量关系与上下文知识,构成一个‘内部世界’。本文提出一种基于Gemma 3的模块化架构,利用该内部世界进行野火发生预测。不依赖Gemma 3原始嵌入和位置编码堆栈,而是设计自定义前馈模块,将表格型野火特征转换为Gemma 3中层Transformer块所需的隐藏维度。冻结这些子层以保留其预训练表征能力,仅训练较小的输入与输出网络。该方法显著减少可训练参数,降低在有限野火数据上的过拟合风险,同时保留Gemma 3的广泛知识优势。在摩洛哥野火数据集上的评估表明,其预测准确率与鲁棒性优于标准前馈与卷积基线。消融实验确认冻结的Transformer层持续贡献更优表征,验证了复用大模型中层作为学习内世界的可能性。研究结果表明,有策略地模块化重用预训练Transformer,可为野火风险管理等关键环境应用提供更高效、可解释的解决方案。

原文摘要 · Abstract (English)

Deep learning models, especially large Transformers, carry substantial "memory" in their intermediate layers -- an \emph{internal world} that encodes a wealth of relational and contextual knowledge. This work harnesses that internal world for wildfire occurrence prediction by introducing a modular architecture built upon Gemma 3, a state-of-the-art multimodal model. Rather than relying on Gemma 3's original embedding and positional encoding stacks, we develop a custom feed-forward module that transforms tabular wildfire features into the hidden dimension required by Gemma 3's mid-layer Transformer blocks. We freeze these Gemma 3 sub-layers -- thus preserving their pretrained representation power -- while training only the smaller input and output networks. This approach minimizes the number of trainable parameters and reduces the risk of overfitting on limited wildfire data, yet retains the benefits of Gemma 3's broad knowledge. Evaluations on a Moroccan wildfire dataset demonstrate improved predictive accuracy and robustness compared to standard feed-forward and convolutional baselines. Ablation studies confirm that the frozen Transformer layers consistently contribute to better representations, underscoring the feasibility of reusing large-model mid-layers as a learned internal world. Our findings suggest that strategic modular reuse of pretrained Transformers can enable more data-efficient and interpretable solutions for critical environmental applications such as wildfire risk management.

野火预测Gemma 3模块化知识复用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。