.png){kind=icon}
返回
Regulating Modality Utilization within Multimodal Fusion Networks
.png){kind=icon}
OA
AI
DOI：10.3390/s24186054.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Multimodal fusion networks play a pivotal role in leveraging diverse sources of information for enhanced machine learning applications in aerial imagery. However, current approaches often suffer from a bias towards certain modalities, diminishing the potential benefits of multimodal data. This paper addresses this issue by proposing a novel modality utilization-based training method for multimodal fusion networks. The method aims to guide the network's utilization on its input modalities, ensuring a balanced integration of complementary information streams, effectively mitigating the overutilization of dominant modalities. The method is validated on multimodal aerial imagery classification and image segmentation tasks, effectively maintaining modality utilization within +/- 10% of the user-defined target utilization and demonstrating the versatility and efficacy of the proposed method across various applications. Furthermore, the study explores the robustness of the fusion networks against noise in input modalities, a crucial aspect in real-world scenarios. The method showcases better noise robustness by maintaining performance amidst environmental changes affecting different aerial imagery sensing modalities. The network trained with 75.0% EO utilization achieves significantly better accuracy (81.4%) in noisy conditions (noise variance = 0.12) compared to traditional training methods with 99.59% EO utilization (73.7%). Additionally, it maintains an average accuracy of 85.0% across different noise levels, outperforming the traditional method's average accuracy of 81.9%. Overall, the proposed approach presents a significant step towards harnessing the full potential of multimodal data fusion in diverse machine learning applications such as robotics, healthcare, satellite imagery, and defense applications.
Keyword:
multimodal
data fusion
modality utilization
permutation feature importance
aerial imagery
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
3.5
论文数:
7.2W
被引数:
20.9W
机构
引用论文
Feature-level fusion approaches based on multimodal EEG data for depression recognition.png)
INFORMATION FUSION
INFORMATION FUSIONIF15.5
PREAI
分享收藏Environment Sound Classification Using a Two-Stream CNN Based on Decision-Level FusionSENSORSOAAI
SENSORSIF3.5
OAAI分享收藏TreeUNet: Adaptive Tree convolutional neural networks for subdecimeter aerial image segmentationTreeUNet: 自适应树卷积神经网络用于亚分米航拍图像分割
PREAI
分享收藏Learning Modality Complementary Features with Mixed Attention Mechanism for RGB-T TrackingSENSORSOAAI
SENSORSIF3.5
OAAI分享收藏
查看更多内容