.png){kind=icon}
返回
Learning Virtual View Selection for 3D Scene Semantic Segmentation
.png){kind=icon}
PRE
AI
DOI：10.1109/TIP.2024.3421952.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
2D-3D joint learning is essential and effective for fundamental 3D vision tasks, such as 3D semantic segmentation, due to the complementary information these two visual modalities contain. Most current 3D scene semantic segmentation methods process 2D images as they are, i.e., only real captured 2D images are used. However, such captured 2D images may be redundant, with abundant occlusion and/or limited field of view (FoV), leading to poor performance for the current methods involving 2D inputs. In this paper, we propose a general learning framework for joint 2D-3D scene understanding by selecting informative virtual 2D views of the underlying 3D scene. We then feed both the 3D geometry and the generated virtual 2D views into any joint 2D-3D-input or pure 3D-input based deep neural models for improving 3D scene understanding. Specifically, we generate virtual 2D views based on an information score map learned from the current 3D scene semantic segmentation results. To achieve this, we formalize the learning of the information score map as a deep reinforcement learning process, which rewards good predictions using a deep neural network. To obtain a compact set of virtual 2D views that jointly cover informative surfaces of the 3D scene as much as possible, we further propose an efficient greedy virtual view coverage strategy in the normal-sensitive 6D space, including 3-dimensional point coordinates and 3-dimensional normal. We have validated our proposed framework for various joint 2D-3D-input or pure 3D-input based deep neural models on two real-world 3D scene datasets, i.e., ScanNet v2 and S3DIS, and the results demonstrate that our method obtains a consistent gain over baseline models and achieves new top accuracy for joint 2D and 3D scene semantic segmentation.
Keyword:
Three-dimensional displays
Semantic segmentation
Solid modeling
Semantics
Geometry
Task analysis
Deep reinforcement learning
Virtual view selection
2D-3D joint learning
deep reinforcement learning
3D semantic segmentation
期刊
IF:
13.7
论文数:
1.0W
被引数:
8.4W
机构
引用论文
.png){kind=icon}
Analysis of the formation of monodisperse populations by homogeneous nucleation通过均相成核分析单分散种群的形成
PREAI
分享收藏Study of bi-directional buck-boost converter topologies for application in electrical vehicle motor drives应用于电动汽车电机驱动的双向buck-boost变换器拓扑研究
PREAI
分享收藏
查看更多内容