CSTR: A Compact Spatio-Temporal Representation for Event-Based Vision

Zaid A. El-Shair

doi:10.1109/ACCESS.2023.3316143

2023-01-01

Zaid A. El-Shair *

Ali Hassani

Samir A. Rawashdeh

DOI：10.1109/ACCESS.2023.3316143

原文链接

原文求助

摘要

En 中文

Event-based vision is a novel perception modality that offers several advantages, such as high dynamic range and robustness to motion blur. In order to process events in batches and utilize modern computer vision deep-learning architectures, an intermediate representation is required. Nevertheless, constructing an effective batch representation is non-trivial. In this paper, we propose a novel representation for event-based vision, called the compact spatio-temporal representation (CSTR). The CSTR encodes an event batch's spatial, temporal, and polarity information in a 3-channel image-like format. It achieves this by calculating the mean of the events' timestamps in combination with the event count at each spatial position in the frame. This representation shows robustness to motion-overlapping, high event density, and varying event-batch durations. Due to its compact 3-channel form, the CSTR is directly compatible with modern computer vision architectures, serving as an excellent choice for deploying event-based solutions. In addition, we complement the CSTR with an augmentation framework that introduces randomized training variations to the spatial, temporal, and polarity characteristics of event data. Experimentation over different object and action recognition datasets shows that the CSTR outperforms other representations of similar complexity under a consistent baseline. Further, the CSTR is made more robust and significantly benefits from the proposed augmentation framework, considerably addressing the sparseness in event-based datasets.

Keyword:

Chemical reactions

Computer vision

Computer architecture

Task analysis

Vehicle dynamics

Object recognition

Visualization

Event detection

Object detection

Data augmentation

Event-based vision