AI-Enabled Cross-Modal Communications

Xin Wei

doi:10.1109/MWC.001.2000448

2021-08-01

PRE

Xin Wei

Liang Zhou *

DOI：10.1109/MWC.001.2000448

原文链接

原文求助

摘要

En 中文

When haptic signals are integrated with existing audio-visual dominated multimedia applications, multi-modal services that can satisfy an individual's immersive experience have emerged. In order to support multi-modal services, cross-modal communications come into being. However, facing collaborative transmission and comprehensive processing requirements of audio, visual, and haptic signals as well as their influences for user's immersive experience, the research of cross-modal communications is still in its infancy and needs to tackle many technical challenges. Due to great successes and powerful abilities of artificial intelligence (AI), it can be expected to underpin cross-modal communications. In this article, we try to deal with issues in cross-modal communications by using AI technology. Specifically, we first adopt the federated learning paradigm to solve sparse data collection and privacy protection problems in the immersive experience description of multi-modal services. Then, we resort to the reinforcement learning paradigm to construct a joint optimization framework of caching, communication, and computation, realizing collaborative transmission of audio, visual, and haptic streams. Finally, we attempt to use the transfer learning paradigm to extract, transfer, and fuse knowledge, semantics, and characteristics from different modalities, recovering corrupted signals and promoting rendering effects at the receiver. Experimental results validate the effectiveness of the AI-enabled cross-modal communications strategies.

Keyword:

Haptic interfaces

Immersive experience

Artificial intelligence

Quality of experience

Visualization

Rendering (computer graphics)

Collaborative work