Event-Aware Video Deraining via Multi-Patch Progressive Learning

doi:10.1109/TIP.2023.3272283

Event-Aware Video Deraining via Multi-Patch Progressive Learning

2023-01-01

9

PRE

AI

DOI：10.1109/TIP.2023.3272283

Original

Original request for help

Save

Abstract

En 中文

In this paper, we address the problem of video-based rain streak removal by developing an event-aware multi-patch progressive neural network. Rain streaks in video exhibit correlations in both temporal and spatial dimensions. Existing methods have difficulties in modeling the characteristics. Based on the observation, we propose to develop a module encoding events from neuromorphic cameras to facilitate deraining. Events are captured asynchronously at pixel-level only when intensity changes by a margin exceeding a certain threshold. Due to this property, events contain considerable information about moving objects including rain streaks passing though the camera across adjacent frames. Thus we suggest that utilizing it properly facilitates deraining performance non-trivially. In addition, we develop a multi-patch progressive neural network. The multi-patch manner enables various receptive fields by partitioning patches and the progressive learning in different patch levels makes the model emphasize each patch level to a different extent. Extensive experiments show that our method guided by events outperforms the state-of-the-art methods by a large margin in synthetic and real-world datasets.

Keywords:

Rain

Cameras

Convolutional neural networks

Task analysis

Neural networks

Correlation

Sun

Video deraining

event-aware

progressive learning

multi-patch learning

Event-Aware Video Deraining via Multi-Patch Progressive Learning

Abstract

Journal

Organization