融合视觉与物理信息,精准估算工业垃圾重量。
Learning to Weigh Waste: A Physics-Informed Multimodal Fusion Framework and Large-Scale Dataset for Commercial and Industrial Applications
- 用图像+尺寸/距离等物理数据,通过注意力机制融合多模态信息。
- 在3.5至3450公斤范围内误差仅88.06公斤,轻物误差低于2.38公斤。
- 可生成可解释的预测理由,适合物流与回收场景落地使用。
准确估计商业与工业垃圾重量对高效运营至关重要,但基于图像的估重仍面临挑战:外观相似物体密度不同,且可见尺寸随相机距离变化。为此,我们提出多模态重量预测框架MWP,结合RGB图像与物理信息(物体尺寸、相机距离、高度)。同时构建了真实世界数据集Waste-Weight-10K,包含10,421组同步采集的图像与元数据,覆盖11类垃圾,重量范围3.5至3,450公斤。模型采用视觉变压器提取视觉特征,专用编码器处理几何与类别信息,通过堆叠互注意力融合机制实现视觉与物理线索相互引导,缓解视角影响并关联材料属性。为保证宽范围性能,采用均方对数误差训练。测试集上,平均绝对误差(MAE)为88.06公斤,平均绝对百分比误差(MAPE)为6.39%,决定系数R²达0.9548。轻物(0–100公斤)误差低至2.38公斤(MAPE 3.1%),重物(1000–2000公斤)仍保持11.1% MAPE。最后引入基于SHAP与大语言模型的物理可解释模块,生成清晰的人类可读解释。
原文摘要 · Abstract (English)
Accurate weight estimation of commercial and industrial waste is important for efficient operations, yet image-based estimation remains difficult because similar-looking objects may have different densities, and the visible size changes with camera distance. Addressing this problem, we propose Multimodal Weight Predictor (MWP) framework that estimates waste weight by combining RGB images with physics-informed metadata, including object dimensions, camera distance, and camera height. We also introduce Waste-Weight-10K, a real-world dataset containing 10,421 synchronized image-metadata collected from logistics and recycling sites. The dataset covers 11 waste categories and a wide weight range from 3.5 to 3,450 kg. Our model uses a Vision Transformer for visual features and a dedicated metadata encoder for geometric and category information, combining them with Stacked Mutual Attention Fusion that allows visual and physical cues guide each other. This helps the model manage perspective effects and link objects to material properties. To ensure stable performance across the wide weight range, we train the model using Mean Squared Logarithmic Error. On the test set, the proposed method achieves 88.06 kg Mean Absolute Error (MAE), 6.39% Mean Absolute Percentage Error (MAPE), and an R2 coefficient of 0.9548. The model shows strong accuracy for light objects in the 0-100 kg range with 2.38 kg MAE and 3.1% MAPE, maintaining reliable performance for heavy waste in the 1000-2000 kg range with 11.1% MAPE. Finally, we incorporate a physically grounded explanation module using Shapley Additive Explanations (SHAP) and a large language model to provide clear, human-readable explanations for each prediction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。