A Multi-Stream Sequence Learning Framework for Human Interaction Recognition

doi:10.1109/THMS.2021.3138708

A Multi-Stream Sequence Learning Framework for Human Interaction Recognition

2022-06-01

16

PRE

AI

DOI：10.1109/THMS.2021.3138708

Original

Original request for help

Save

Abstract

En 中文

Human interaction recognition (HIR) is challenging due to multiple humans' involvement and their mutual interaction in a single frame, generated from their movements. Mainstream literature is based on three-dimensional (3-D) convolutional neural networks (CNNs), processing only visual frames, where human joints data play a vital role in accurate interaction recognition. Therefore, this article proposes a multistream network for HIR that intelligently learns from skeletons' key points and spatiotemporal visual representations. The first stream localises the joints of the human body using a pose estimation model and transmits them to a 1-D CNN and bidirectional long short-term memory to efficiently extract the features of the dynamic movements of each human skeleton. The second stream feeds the series of visual frames to a 3-D convolutional neural network to extract the discriminative spatiotemporal features. Finally, the outputs of both streams are integrated via fully connected layers that precisely classify the ongoing interactions between humans. To validate the performance of the proposed network, we conducted a comprehensive set of experiments on two benchmark datasets, UT-interaction and TV human interaction, and found 1.15% and 10.0% improvement in the accuracy.

Keywords:

Feature extraction

Skeleton

Pose estimation

Computer architecture

Computational modeling

Logic gates

Optical imaging

Bidirectional LSTM (BD-LSTM)

three-dimensional (3-D) convolutional neural network (CNN)

1-D CNN

human interaction recognition (HIR)

human pose estimation

skeleton joint key points

multistream network

A Multi-Stream Sequence Learning Framework for Human Interaction Recognition

Abstract

Journal

Organization