Edge-Aware Network for Flow-Based Video Frame Interpolation

doi:10.1109/TNNLS.2022.3178281

2024-01-01

PRE

Bin Zhao

Xuelong Li *

DOI：10.1109/TNNLS.2022.3178281

原文链接

原文求助

摘要

En 中文

Video frame interpolation can up-convert the frame rate and enhance the video quality. In recent years, although interpolation performance has achieved great success, image blur usually occurs at object boundaries owing to the large motion. It has been a long-standing problem and has not been addressed yet. In this brief, we propose to reduce the image blur and get the clear shape of objects by preserving the edges in the interpolated frames. To this end, the proposed edge-aware network (EA-Net) integrates the edge information into the frame interpolation task. It follows an end-to-end architecture and can be separated into two stages, i.e., edge-guided flow estimation and edge-protected frame synthesis. Specifically, in the flow estimation stage, three edge-aware mechanisms are developed to emphasize the frame edges in estimating flow maps, so that the edge maps are taken as auxiliary information to provide more guidance to boost the flow accuracy. In the frame synthesis stage, the flow refinement module is designed to refine the flow map, and the attention module is carried out to adaptively focus on the bidirectional flow maps when synthesizing the intermediate frames. Furthermore, the frame and edge discriminators are adopted to conduct the adversarial training strategy, so as to enhance the reality and clarity of synthesized frames. Experiments on three benchmarks, including Vimeo90k, UCF101 for single-frame interpolation, and Adobe240-fps for multiframe interpolation, have demonstrated the superiority of the proposed EA-Net for the video frame interpolation task.

Keyword:

Interpolation

Estimation

Task analysis

Kernel

Image edge detection

Computer architecture

Training

Adversarial training

edge-aware

flow estimation

video frame interpolation