TASK AWARE MULTI-TASK LEARNING FOR SPEECH TO TEXT TASKS
In general, the direct Speech-to-text translation (ST) is jointly trained with Automatic Speech Recognition (ASR), and Machine Translation (MT) tasks. However, the issues with the current joint learning strategies inhibit the knowledge transfer across these tasks. We propose a task modulation network which allows the model to learn task specific features, while learning the shared features simultaneously. This proposed approach removes the need for separate finetuning step resulting in a single model which performs all these tasks. This single model achieves a performance of 28.64 BLEU score on ST MuST-C English-German, WER of 11.61% on ASR TEDLium v3, 23.35 BLEU score on MT WMT’15 English-German task. This sets a new state-of-the-art performance (SOTA) on the ST task while outperforming the existing end-to-end ASR systems.
PDF Abstract
LibriSpeech
OpenSubtitles
MuST-C
Results from the Paper
Ranked #1 on
Speech-to-Text Translation
on MuST-C EN->DE
(using extra training data)
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Uses Extra Training Data |
Benchmark |
|---|---|---|---|---|---|---|---|
| Speech-to-Text Translation | MuST-C EN->DE | Task Modulation + Multitask Learning(ASR/MT) + Data Augmentation | Case-sensitive sacreBLEU | 28.88 | # 1 |
