.png){kind=icon}
返回
Structure-Aware Cross-Modal Transformer for Depth Completion
.png){kind=icon}
PRE
AI
DOI：10.1109/TIP.2024.3355807.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
In this paper, we present a Structure-aware Cross-Modal Transformer (SCMT) to fully capture the 3D structures hidden in sparse depths for depth completion. Most existing methods learn to predict dense depths by taking depths as an additional channel of RGB images or learning 2D affinities to perform depth propagation. However, they fail to exploit 3D structures implied in the depth channel, thereby losing the informative 3D knowledge that provides important priors to distinguish the foreground and background features. Moreover, since these methods rely on the color textures of 2D images, it is challenging for them to handle poor-texture regions without the guidance of explicit 3D cues. To address this, we disentangle the hierarchical 3D scene-level structure from the RGB-D input and construct a pathway to make sharp depth boundaries and object shape outlines accessible to 2D features. Specifically, we extract 2D and 3D features from depth inputs and the back-projected point clouds respectively by building a two-stream network. To leverage 3D structures, we construct several cross-modal transformers to adaptively propagate multi-scale 3D structural features to the 2D stream, energizing 2D features with priors of object shapes and local geometries. Experimental results show that our SCMT achieves state-of-the-art performance on three popular outdoor (KITTI) and indoor (VOID and NYU) benchmarks.
Keyword:
Depth completion
cross-modal interaction
structure learning
transformer
期刊
IF:
13.7
论文数:
1.0W
被引数:
8.4W
机构
引用论文
LOCATING STATIONS OF PUBLIC TRANSPORTATION VEHICLES FOR IMPROVING TRANSIT ACCESSIBILITY.png)
TRANSPORT
OAAI
TRANSPORTIF0
OAAI分享收藏分享收藏
查看更多内容