提出首个立体转换基准数据集,揭示现有方法缺陷并改进评估与模型。
Mono2Stereo: A Benchmark and Empirical Study for Stereo Conversion
- 构建高质量单图转立体数据集,支持系统性研究
- 发现现有评价指标忽略关键视差区域,导致误判
- 提出新评估指标与统一优化模型,显著提升立体效果
随着3D设备普及和3D内容短缺,立体转换日益受到关注。近期工作将预训练扩散模型(DMs)引入该任务,但受限于大规模训练数据和全面基准,如何有效使用DMs及准确评估立体效果仍不明确。本文提出Mono2Stereo数据集,提供高质量训练数据与评测基准,支持深入研究。基于此,我们开展实证分析,得出两大发现:1)左右视图差异细微,但现有指标对整体像素敏感,未聚焦影响立体感知的关键区域;2)主流方法采用单阶段左至右生成或变形-修补流程,分别面临立体感下降和图像失真问题。据此,我们提出新的评价指标——立体交并比(Stereo IoU),强调视差信息,与人类判断高度相关。同时提出一个强基线模型,在保持图像质量的同时协同优化立体效果,显著优于当前主流方法。代码与数据将开源,模型可访问mono2stereo-bench.github.io。
原文摘要 · Abstract (English)
With the rapid proliferation of 3D devices and the shortage of 3D content, stereo conversion is attracting increasing attention. Recent works introduce pretrained Diffusion Models (DMs) into this task. However, due to the scarcity of large-scale training data and comprehensive benchmarks, the optimal methodologies for employing DMs in stereo conversion and the accurate evaluation of stereo effects remain largely unexplored. In this work, we introduce the Mono2Stereo dataset, providing high-quality training data and benchmark to support in-depth exploration of stereo conversion. With this dataset, we conduct an empirical study that yields two primary findings. 1) The differences between the left and right views are subtle, yet existing metrics consider overall pixels, failing to concentrate on regions critical to stereo effects. 2) Mainstream methods adopt either one-stage left-to-right generation or warp-and-inpaint pipeline, facing challenges of degraded stereo effect and image distortion respectively. Based on these findings, we introduce a new evaluation metric, Stereo Intersection-over-Union, which prioritizes disparity and achieves a high correlation with human judgments on stereo effect. Moreover, we propose a strong baseline model, harmonizing the stereo effect and image quality simultaneously, and notably surpassing current mainstream methods. Our code and data will be open-sourced to promote further research in stereo conversion. Our models are available at mono2stereo-bench.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。