Spatial-temporal 3D dependency matching with self-supervised deep learning for monocular visual sensing

Chengqun Song

doi:10.1016/j.neucom.2022.01.074

Spatial-temporal 3D dependency matching with self-supervised deep learning for monocular visual sensing

2022-04-01

6

PRE

AI

DOI：10.1016/j.neucom.2022.01.074

Original

Original request for help

Save

Abstract

En 中文

Monocular visual sensing is the task of using a camera to estimate the scene depth, optical flow and camera pose. In this paper, we propose a spatial-temporal 3D dependency matching approach that enforces the robustness of continuous frames matching for monocular visual sensing. 3D structure and warped depth based geometry backpropagation are used to encourage jointly learning the view depth, optical flow and camera pose employing a novel self-supervised neural network from monocular sequences. We designed two different iterative convolutional prediction sub-networks, where the optical flow obtained by depth and camera pose is iteratively used for depth prediction. A virtual frame method is proposed to optimize the optical flow of moving objects. The salient feature of the proposed learning framework is completely unsupervised, requiring only consecutive monocular images for training and testing. Evaluation on publicly benchmark datasets shows that our unsupervised learning model significantly outperforms previous methods and achieves better performance compared with previously unsupervised manners and achieves comparable results with supervised ones. (c) 2022 Elsevier B.V. All rights reserved.

Keywords:

Self-supervised deep learning

Depth estimation

Optical flow

Camera pose

Monocular vision

Spatial-temporal 3D dependency matching with self-supervised deep learning for monocular visual sensing

Abstract

Journal

Organization