神经网络中的场景坐标回归会泄露隐私,攻击者可重建真实环境3D结构。
Seeing Through the Weights: Privacy Leakage in Scene Coordinate Regression

- 通过批量查询模型获取像素级3D坐标,利用稳定性筛选可靠点
- 在室内和室外数据集上实现高保真度的3D场景重建,几何精度达90%以上
- 不仅能恢复结构,还能反演生成任意视角图像,暴露敏感视觉信息
场景坐标回归(SCR)方法被广泛用于视觉定位,其将场景隐式编码于神经网络参数中,不存储图像或地图,常被认为具备隐私保护性。本文揭示该假设在实践中不成立:提出一种基于查询的攻击方法,仅需对目标模型进行不同层级访问,即可从无关联的代理图像批量查询中提取密集像素级3D坐标。通过小扰动下坐标稳定性判断可靠点,并在白盒条件下进一步优化。这些稳定点跨批次累积后,可重构训练环境的3D几何结构。此外,从恢复的表示中反演网络特征,能合成任意视角图像,还原近似颜色外观。实验表明,在室内与室外数据集上,大量训练环境可实现高几何保真度重建。除几何外,还恢复了可辨识的布局与潜在敏感元素,直接挑战文献中关于SCR天然隐私保护的断言。该发现警示此类系统在私密或安全关键场景部署时的真实风险。
原文摘要 · Abstract (English)
Scene Coordinate Regression (SCR) methods are increasingly adopted for visual localization. In these approaches, the scene is implicitly encoded within a neural network that regresses a 3D world coordinate for each image pixel. Because the scene is represented only through the network parameters and not stored explicitly as images or maps, such methods are often assumed to be privacy-preserving. In this work, we show that this assumption is incorrect in practice. Specifically, we introduce a query-based attack that reconstructs the 3D geometry of the training environment from an SCR model under different levels of model access. To do so, we repeatedly query the model with batches of proxy images unrelated to the target scene to obtain dense pixel-wise 3D coordinates. Reliable points are identified through their stability under small input perturbations and can be further refined in a white-box setting. These stable points are accumulated across independent query batches to recover the scene geometry. From the recovered 3D representation, we also invert the network features to synthesize images from arbitrary viewpoints, revealing additional appearance information. Experiments on indoor and outdoor datasets demonstrate that substantial portions of training environments can be reconstructed with high geometric fidelity. Beyond geometry, we also recover an approximate color appearance, which exposes recognizable layout and potentially sensitive scene elements. This directly contradicts claims in the literature that SCR representations are privacy-preserving by design, and reveals a real risk when such systems are deployed in private or security-critical spaces. The project page is available at https://jaeminch0.github.io/seeing-through-the-weights-privacy-leakage-in-scene-coordinate-regression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。