Self-Supervised Learning With Segmental Masking for Speech Representation

Xianghu Yue

doi:10.1109/JSTSP.2022.3191845

2022-10-01

Xianghu Yue *

Jingru Lin

Haizhou Li

DOI：10.1109/JSTSP.2022.3191845

原文链接

原文求助

摘要

En 中文

Self-supervised learning has achieved remarkable success for learning speech representations from unlabeled data. The masking strategy plays an important role in the self-supervised learning algorithm. Most of the masking techniques operate at a frame level. In linguistics, phone is the smallest unit of sound. Hence, we believe that a masking technique that operates at a phoneme level will effectively encode the phonotactic and prosodic constraints of a spoken language, thus eventually benefits the downstream speech recognition tasks. In this work, we explore a novel segmental masking strategy. Specifically, we mask phonetically motivated speech segments according to the phonetic segmentation in an utterance. By doing so, we implicitly incorporate the properties of a spoken language, such as phonotactic constraints and duration of phonetic segments, into the pre-training. Through extensive experiments, we confirm that the segmental masking strategy consistently outperforms the frame-based masking counterpart. We also further investigate the effect of segmental masking unit size, i.e. phoneme, phoneme span, and lexical word. This work presents an important finding about masking strategy in speech representation learning.

Keyword:

Task analysis

Speech processing

Self-supervised learning

Speech recognition

Representation learning

Supervised learning

Data models

Self-supervised learning

speech representation learning

segmental masking