.png){kind=icon}
返回
Knowledge-driven Egocentric Multimodal Activity Recognition
.png){kind=icon}
PRE
AI
DOI：10.1145/3409332.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Recognizing activities from egocentric multimodal data collected by wearable cameras and sensors, is gaining interest, as multimodal methods always benefit from the complementarity of different modalities. However, since high-dimensional videos contain rich high-level semantic information while low-dimensional sensor signals describe simple motion patterns of the wearer, the large modality gap between the videos and the sensor signals raises a challenge for fusing the raw data. Moreover, the lack of large-scale egocentric multimodal datasets due to the cost of data collection and annotation processes makes another challenge for employing complex deep learning models. To jointly deal with the above two challenges, we propose a knowledge-driven multimodal activity recognition framework that exploits external knowledge to fuse multimodal data and reduce the dependence on large-scale training samples. Specifically, we design a dual-GCLSTM (Graph Convolutional LSTM) and a multi-layer GCN (Graph Convolutional Network) to collectively model the relations among activities and intermediate objects. The dual-GCISTM is designed to fuse temporal multimodal features with top-down relation-aware guidance. In addition, we apply a co-attention mechanism to adaptively attend to the features of different modalities at different timesteps. The multi-layer GCN aims to learn relation-aware classifiers of activity categories. Experimental results on three publicly available egocentric multimodal datasets show the effectiveness of the proposed model.
Keyword:
Egocentric videos
wearable sensors
graph neural networks
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
6
论文数:
2.0K
被引数:
5.4K
机构
引用论文
Alterations of regional cerebral blood flow and oxygen metabolism in Parkinson's disease.png)
Neurology
NeurologyIF0
PREAI
分享收藏Late diagenesis of illite-smectite in the Podhale Basin, southern Poland: Chemistry, morphology, and preferred orientationGeosphere
OAAI
GeosphereIF0
OAAI分享收藏Long-term efficacy of goniosynechialysis combined with phacoemulsification for primary angle closure
PREAI
分享收藏
查看更多内容