Joint Deep-Unfolding Optimization Learning for Depth Map Arbitrary-Scale Super-Resolution

J. W. Zhang

doi:10.1109/TMM.2025.3613083

Joint Deep-Unfolding Optimization Learning for Depth Map Arbitrary-Scale Super-Resolution

2025-01-01

1

PRE

AI

DOI：10.1109/TMM.2025.3613083

Original

Original request for help

Save

Abstract

En 中文

Color-guided Depth map Super-Resolution (DSR) based on Convolutional Neural Networks (CNN) is a crucial technology to remedy the defects of mainstream commercial depth cameras and has made significant progress in recent years. Nevertheless, this technology is inevitably facing some huge challenges. Firstly, existing CNN-based DSR methods are designed as black-box network architectures. Secondly, few approaches study single model to achieve arbitrary-scale DSR. Thirdly, due to structural inconsistency between dual-modality, color-guided DSR methods always face texture-copying issue. To this end, we propose a novel joint DSR and high-low frequency decomposition optimization model and this model is unfolded into Deep Arbitrary-Scale Unfolding Network (DASU-Net). DASU-Net can achieve robust continuous representation ability by alternately-iterative updating of high-low frequencies and depth features. More importantly, Arbitrary-scale Up-sampling Fusion (AUF) module is proposed to achieve arbitrary-scale up-sampling and dual-modality feature fusion. Specifically, two essential components make up the cores of AUF module including arbitrary-scale up-sampling block as well as Feature Enhancement and Multiple Strategies Fusion (FEMSF) blocks. In FEMSF block, color features are first enhanced to highlight its inherently-correlated structure with the guidance of depth features, and then the enhanced features are modulated according to different fusion strategies. Furthermore, a fast version of DASU-Net is proposed to fit real-time scenes, named FDASU-Net, which can diminish the runtime by several times for a depth map with a size of 640 x 480 during inference. A large number of experiments can demonstrate that our DASU-Net and FDASU-Net can transcend many state-of-the-art DSR methods in terms of several quantitative and qualitative indexes.

Keywords:

Optimization models

Image reconstruction

Convolutional neural networks

Superresolution

Kernel

Interpolation

Color

Training

Real-time systems

Convolution

High-low frequency decomposition

depth map super-resolution

arbitrary-scale up-sampling

explainable network

Joint Deep-Unfolding Optimization Learning for Depth Map Arbitrary-Scale Super-Resolution

Abstract

Journal

Organization