A large-scale combinatorial benchmark for sign language recognition

doi:10.1016/j.patcog.2024.111246

A large-scale combinatorial benchmark for sign language recognition

2025-05-01

0

PRE

AI

DOI：10.1016/j.patcog.2024.111246

Original

Original request for help

Save

Abstract

En 中文

Lacking a large-scale dataset is the major obstacle limiting sign language recognition (SLR) to work well in the real world, because of the huge collection and annotation cost of sign language videos. This paper rethinks a sign language sentence as a combination of a template (T) and an entity (E) and presents a novel T and E Disassemble-and-reAssemble (TEDA) strategy to collect T and E sign videos independently. The proposed TEDA strategy has a theoretical capability of generating T x E effective samples with only T + E collection and annotation cost. With the TEDA strategy, we build a cost-controllable large-scale (CCLS) sign language dataset with 300,400 combinatorial samples, generated from 6,000 T videos and 29,700 E videos. To enable training arbitrary SLR models on combinatorial data, we propose a combinatorial SLR framework. Specifically, we first design a dynamic combination module to dynamically combine independent T and E features to generate combinatorial features. Then, we propose a joint constraint module to ensure that the distribution of the combinatorial features is as close as possible with the complete features. Finally, we develop a multi-stage training strategy to accommodate SLR learning with the combinatorial data. Plentiful experiments demonstrate the rationality of our TEDA strategy in generating large-scale effective combinatorial samples as well as the effectiveness of the combinatorial framework in promoting SLR.

Keywords:

Sign language recognition

T and E disassemble-and-reassemble strategy

Cost-controllable large-scale dataset

Combinatorial framework

A large-scale combinatorial benchmark for sign language recognition

Abstract

Journal

Organization