Grounded Sequence to Sequence Transduction

doi:10.1109/JSTSP.2020.2998415

Grounded Sequence to Sequence Transduction

2020-03-01

0

PRE

AI

DOI：10.1109/JSTSP.2020.2998415

Original

Original request for help

Save

Abstract

En 中文

Speech recognition and machine translation have made major progress over the past decades, providing practical systems to map one language sequence to another. Although multiple modalities such as sound and video are becoming increasingly available, the state-of-the-art systems are inherently unimodal, in the sense that they take a single modality - either speech or text - as input. Evidence from human learning suggests that additional modalities can provide disambiguating signals crucial for many language tasks. In this article, we describe the How2 dataset , a large, open-domain collection of videos with transcriptions and their translations. We then show how this single dataset can be used to develop systems for a variety of language tasks and present a number of models meant as starting points. Across tasks, we find that building multimodal architectures that perform better than their unimodal counterpart remains a challenge. This leaves plenty of room for the exploration of more advanced solutions that fully exploit the multimodal nature of the How2 dataset , and the general direction of multimodal learning with other datasets as well.

Keywords:

Visualization

Feature extraction

Speech recognition

Task analysis

Training

Adaptation models

Grounding

multimodal machine learning

speech recognition

machine translation

representation learning

summarization

Grounded Sequence to Sequence Transduction

Abstract

AI Summary

Journal

Organization