Modality Conversion Meets Superresolution: A Collaborative Framework for High- Resolution Thermal UAV Image Generation

Zhicheng Zhao

doi:10.1109/TGRS.2024.3354878

2024-01-01

PRE

DOI：10.1109/TGRS.2024.3354878

原文链接

原文求助

摘要

En 中文

Due to the limitations and costs of thermal sensors, unmanned aerial vehicle (UAV) platforms often equip with high-resolution (HR) visible imaging and low-resolution (LR) thermal imaging cameras for all-day monitoring capability. Existing works generate the HR thermal UAV images by either superresolution (SR) from HR visible and LR thermal images or modality conversion (MC) from HR visible images. However, the modality gap between visible and thermal sources might degrade the generation quality. We observe that the MC task is beneficial in addressing the cross-modal gap in the SR task, while the SR task can provide the condition of thermal information to boost the MC task. Moreover, these two tasks have the same output and can thus be carried out simultaneously without any additional annotation. Based on this observation, we propose a collaborative enhancement network (CENet), which performs thermal UAV image SR and visible image MC in a joint manner, for HR thermal UAV image generation. In particular, we design a mutual guidance module (MGM) to interact the features from SR and MC tasks in an alternating bidirectional manner. Considering that low-level vision tasks are position-sensitive, to further enhance the feature alignment between the two tasks, we design a bidirectional alignment fusion module (BAFM) to maintain feature consistency of the MC and SR branches. The proposed collaborative framework not only achieves joint and unified training of the two tasks, but also generates two types of complementary HR images. Extensive experiments on public datasets demonstrate that the proposed CENet outperforms current state-of-the-art SR methods in generating HR thermal UAV images, as quantified by peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM).

Keyword:

Task analysis

Superresolution

Autonomous aerial vehicles

Feature extraction

Remote sensing

Imaging

Collaboration

Collaborative learning

modality conversion (MC)

remote sensing

thermal image superresolution (SR)