.png){kind=icon}
.png){kind=icon}
Return
Fusing Cross-Domain Knowledge from Multimodal Data to Solve Problems in the Physical World
.png){kind=icon}
PRE
AI
Z
DOI：10.1145/3768625.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
The proliferation of artificial intelligence has enabled a diversity of applications that bridge the gap between digital and physical worlds. As physical environments are too complex to model through a single information acquisition approach, it is crucial to fuse multimodal data generated by different sources, such as sensors, devices, systems, and people, to solve a problem in the real world. Unfortunately, it is neither applicable nor sustainable to deploy new resources to collect original data from scratch for every problem. Thus, when data is inadequate in the domain of the problem, it is vital to fuse knowledge from multimodal data that is already available in other domains. We call this cross-domain knowledge fusion. Existing research focuses on fusing multimodal data in a single domain, supposing the knowledge from different datasets is intrinsically aligned; however, this assumption may not hold in the scenarios of cross-domain knowledge fusion. In this article, we formally define the cross-domain multimodal data fusion problem, discussing its unique challenges, differences, and advantages beyond data fusion in a single domain. We propose a four-layer framework, consisting of Domains, Links, Models, and Data layers, answering three key questions: what to fuse, why can be fused, and how to fuse. The Domains Layer selects relevant data from different domains for a given problem. The Links Layer reveals the philosophy of knowledge alignment beyond specific model structures. The Models Layer provides two knowledge fusion paradigms based on the fundamental mechanisms for processing data. The Data Layer turns data of different structures, resolutions, scales, and distributions into a consistent representation that can be fed into an AI model. With this framework, we can design solutions that fuse cross-domain multimodal data effectively for solving real-world problems.
Keywords:
Knowledge fusion
data fusion
multimodal data
cross-domain
data ransformation
urban computing
spatio-temporal data
deep learning
Journal
IF:
6.6
Papers:
1.5K
Citations:
6.2K
Organization
Cited Papers