AutoEncoder-Driven Multimodal Collaborative Learning for Medical Image Synthesis

doi:10.1007/s11263-023-01791-0

AutoEncoder-Driven Multimodal Collaborative Learning for Medical Image Synthesis

2023-05-03

8

PRE

AI

DOI：10.1007/s11263-023-01791-0

Original

Original request for help

Save

Abstract

En 中文

Multimodal medical images have been widely applied in various clinical diagnoses and treatments. Due to the practical restrictions, certain modalities may be hard to acquire, resulting in incomplete data. Existing methods attempt to generate the missing data with multiple available modalities. However, the modality differences in tissue contrast and lesion appearance become an obstacle to making a precise estimation. To address this issue, we propose an autoencoder-driven multimodal collaborative learning framework for medical image synthesis. The proposed approach takes an autoencoder to comprehensively supervise the synthesis network using the self-representation of target modality, which provides target-modality-specific prior to guide multimodal image fusion. Furthermore, we endow the autoencoder with adversarial learning capabilities by converting its encoder into a pixel-sensitive discriminator capable of both reconstruction and discrimination. To this end, the generative model is completely supervised by the autoencoder. Considering the efficiency of multimodal generation, we also introduce a modality mask vector as the target modality label to guide the synthesis direction, empowering our method to estimate any missing modality with a single model. Extensive experiments on multiple medical image datasets demonstrate the significant generalization capability as well as the superior synthetic quality of the proposed method, compared with other competing methods.

Keywords:

Multimodal collaborative learning

Medical image synthesis

Autoencoder

Self-representation

AutoEncoder-Driven Multimodal Collaborative Learning for Medical Image Synthesis

Abstract

Journal

Organization