76 papers with code • 2 benchmarks • 2 datasets
Pre-training a neural network using unsupervised (self-supervised) auxiliary tasks on unlabeled data.
Our experiments on WSJ reduce WER of a strong character-based log-mel filterbank baseline by up to 36% when only a few hours of transcribed data is available.
We further exhibit a new class of random orthogonal initial conditions on weights that, like unsupervised pre-training, enjoys depth independent learning times.
We show that constituency parsing benefits from unsupervised pre-training across a variety of languages and a range of pre-training conditions.
In this paper, we compose a trilogy of exploring the basic and generic supervision in the sequence from spatial, spatiotemporal and sequential perspectives.
Our representations are learned using a contrastive loss, where two augmented clips from the same short video are pulled together in the embedding space, while clips from different videos are pushed away.
In this work we propose for the first time a transformer-based framework for unsupervised representation learning of multivariate time series.
Self-training and unsupervised pre-training have emerged as effective approaches to improve speech recognition systems using unlabeled data.