Search Results for author:
Found 72 papers, 35 papers with code
Deep convolutional acoustic word embeddings using word-pair side information
Recent studies have been revisiting whole words as the basic modelling unit in speech recognition and query applications, instead of phonetic units.
Unsupervised word segmentation and lexicon discovery using acoustic word embeddings
In settings where only unlabelled speech data is available, speech technology needs to be developed without transcriptions, pronunciation dictionaries, or language modelling text.
A segmental framework for fully-unsupervised large-vocabulary speech recognition
We also show that the discovered clusters can be made less speaker- and gender-specific by using an unsupervised autoencoder-like feature extractor to learn better frame-level features (prior to embedding).
Weakly supervised spoken term discovery using cross-lingual side information
Recent work on unsupervised term discovery (UTD) aims to identify and cluster repeated word-like units from audio alone.
Unsupervised neural and Bayesian models for zero-resource speech processing
Finally, we show that the clusters discovered by the segmental Bayesian model can be made less speaker- and gender-specific by using features from the cAE instead of traditional acoustic features.
Towards speech-to-text translation without speech recognition
We explore the problem of translating speech to text in low-resource scenarios where neither automatic speech recognition (ASR) nor machine translation (MT) are available, but we have training data in the form of audio paired with text translations.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
An embedded segmental K-means model for unsupervised segmentation and clustering of speech
Unsupervised segmentation and clustering of unlabelled speech are core problems in zero-resource speech processing.
Visually grounded learning of keyword prediction from untranscribed speech
In this setting of images paired with untranscribed spoken captions, we consider whether computer vision systems can be used to obtain textual labels for the speech.
Query-by-Example Search with Discriminative Neural Acoustic Word Embeddings
Query-by-example search often uses dynamic time warping (DTW) for comparing queries and proposed matching segments.
Semantic speech retrieval with a visually grounded model of untranscribed speech
We introduce a newly collected data set of human semantic relevance judgements and an associated task, semantic speech retrieval, where the goal is to search for spoken utterances that are semantically relevant to a given text query.
Low-Resource Speech-to-Text Translation
We explore models trained on between 20 and 160 hours of data, and find that although models trained on less data have considerably lower BLEU scores, they can still predict words with relatively high precision and recall---around 50% for a model trained on 50 hours of data, versus around 60% for the full 160 hour model.
Visually grounded cross-lingual keyword spotting in speech
Recent work considered how images paired with speech can be used as supervision for building speech systems when transcriptions are not available.
Learning Dynamics of Linear Denoising Autoencoders
Here we develop theory for how noise influences learning in DAEs.
Fast ASR-free and almost zero-resource keyword spotting using DTW and CNNs for humanitarian monitoring
While the resulting CNN keyword spotter cannot match the performance of the DTW-based system, it substantially outperforms a CNN classifier trained only on the keywords, improving the area under the ROC curve from 0. 54 to 0. 64.
ASR-free CNN-DTW keyword spotting using multilingual bottleneck features for almost zero-resource languages
We consider multilingual bottleneck features (BNFs) for nearly zero-resource keyword spotting.
Pre-training on high-resource speech recognition improves low-resource speech-to-text translation
Finally, we show that the approach improves performance on a true low-resource task: pre-training on a combination of English ASR and French ASR improves Mboshi-French ST, where only 4 hours of data are available, from 3. 5 to 7. 1
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder models
We investigate unsupervised models that can map a variable-duration speech segment to a fixed-dimensional representation.
Critical initialisation for deep signal propagation in noisy rectifier neural networks
Simulations and experiments on real-world data confirm that our proposed initialisation is able to stably propagate signals in deep networks, while using an initialisation disregarding noise fails to do so.
Multimodal One-Shot Learning of Speech and Images
Imagine a robot is shown new concepts visually together with spoken tags, e. g. "milk", "eggs", "butter".
Multilingual and Unsupervised Subword Modeling for Zero-Resource Languages
Here we directly compare multiple methods, including some that use only target language speech data and some that use transcribed speech from other (non-target) languages, and we evaluate using two intrinsic measures as well as on a downstream unsupervised word segmentation and clustering task.
Feature exploration for almost zero-resource ASR-free keyword spotting using a multilingual bottleneck extractor and correspondence autoencoders
We compare features for dynamic time warping (DTW) when used to bootstrap keyword spotting (KWS) in an almost zero-resource setting.
Semantic query-by-example speech search using visual grounding
A number of recent studies have started to investigate how speech systems can be trained on untranscribed speech by leveraging accompanying images at training time.
Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks
For our submission to the ZeroSpeech 2019 challenge, we apply discrete latent-variable neural networks to unlabelled speech and use the discovered units for speech synthesis.
On the Contributions of Visual and Textual Supervision in Low-Resource Semantic Speech Retrieval
Recent work has shown that speech paired with images can be used to learn semantically meaningful speech representations even without any textual supervision.
Cross-lingual topic prediction for speech using translations
Given a large amount of unannotated speech in a low-resource language, can we classify the speech utterances by topic?
On the expected behaviour of noise regularised deep neural networks as Gaussian processes
Recent work has established the equivalence between deep neural networks and Gaussian processes (GPs), resulting in so-called neural network Gaussian processes (NNGPs).
If dropout limits trainable depth, does critical initialisation still matter? A large-scale statistical analysis on ReLU networks
Our results therefore suggest that, in the shallow-to-moderate depth setting, critical initialisation provides zero performance gains when compared to off-critical initialisations and that searching for off-critical initialisations that might improve training speed or generalisation, is likely to be a fruitless endeavour.
BINet: a binary inpainting network for deep patch-based image compression
We propose the Binary Inpainting Network (BINet), an autoencoder framework which incorporates binary inpainting to reinstate interdependencies between adjacent patches, for improved patch-based compression of still images.
Deep motion estimation for parallel inter-frame prediction in video compression
Standard video codecs rely on optical flow to guide inter-frame prediction: pixels from reference frames are moved via motion vectors to predict target video frames.
Multilingual acoustic word embedding models for processing zero-resource languages
Acoustic word embeddings are fixed-dimensional representations of variable-length speech segments.
Unsupervised feature learning for speech using correspondence and Siamese networks
Dynamic programming is then used to align the feature frames between each word pair, serving as weak top-down supervision for the two models.
Analyzing autoencoder-based acoustic word embeddings
To better understand the applications of AWEs in various downstream tasks and in cognitive modeling, we need to analyze the representation spaces of AWEs.
Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge
The idea is to learn a representation of speech by predicting future acoustic units.
Ranked #1 on
Voice Conversion
on ZeroSpeech 2019 English
(using extra training data)
Improved acoustic word embeddings for zero-resource languages using multilingual transfer
We consider three multilingual recurrent neural network (RNN) models: a classifier trained on the joint vocabularies of all training languages; a Siamese RNN trained to discriminate between same and different words from multiple languages; and a correspondence autoencoder (CAE) RNN trained to reconstruct word pairs.
Evaluating computational models of infant phonetic learning across languages
In the first year of life, infants' speech perception becomes attuned to the sounds of their native language.
Unsupervised vs. transfer learning for multimodal one-shot matching of speech and images
Here we compare transfer learning to unsupervised models trained on unlabelled in-domain data.
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages
Research in NLP lacks geographic diversity, and the question of how NLP can be scaled to low-resourced languages has not yet been adequately solved.
A Correspondence Variational Autoencoder for Unsupervised Acoustic Word Embeddings
We propose a new unsupervised model for mapping a variable-duration speech segment to a fixed-dimensional representation.
Direct multimodal few-shot learning of speech and images
We propose direct multimodal few-shot models that learn a shared embedding space of spoken words and images from only a few paired examples.
Towards localisation of keywords in speech using weak supervision
Of the two forms of supervision, the visually trained model performs worse than the BoW-trained model.
Towards unsupervised phone and word segmentation using self-supervised vector-quantized neural networks
We specifically constrain pretrained self-supervised vector-quantized (VQ) neural networks so that blocks of contiguous feature vectors are assigned to the same code, thereby giving a variable-rate segmentation of the speech into discrete units.
A comparison of self-supervised speech representations as input features for unsupervised acoustic word embeddings
We compare frame-level features from contrastive predictive coding (CPC), autoregressive predictive coding and a CAE to conventional MFCCs.
A phonetic model of non-native spoken word processing
We then test the model on a spoken word processing task, showing that phonology may not be necessary to explain some of the word processing effects observed in non-native speakers.
Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptation
We consider how a recent contrastive learning loss can be used in both the purely unsupervised and multilingual transfer settings.
StarGAN-ZSVC: Towards Zero-Shot Voice Conversion in Low-Resource Contexts
We specifically extend the recent StarGAN-VC model by conditioning it on a speaker embedding (from a potentially unseen speaker).
Attention-Based Keyword Localisation in Speech using Visual Grounding
Visually grounded speech models learn from images paired with spoken captions.
Multilingual transfer of acoustic word embeddings improves when training on languages related to the target zero-resource language
Through finer-grained analysis, we show that training on even just a single related language gives the largest gain.
Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processing
In this paper, we first show that the per-utterance mean of CPC features captures speaker information to a large extent.
Feature learning for efficient ASR-free keyword spotting in low-resource languages
We show that, using these features, the CNN-DTW keyword spotter performs almost as well as the DTW keyword spotter while outperforming a baseline CNN trained only on the keyword templates.
A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion
Specifically, we compare discrete and soft speech units as input features.
Voice Conversion Can Improve ASR in Very Low-Resource Settings
In this work we assess whether a VC system can be used cross-lingually to improve low-resource speech recognition.
Towards Learning to Speak and Hear Through Multi-Agent Communication over a Continuous Acoustic Channel
Multi-agent reinforcement learning has been used as an effective means to study emergent communication between agents, yet little focus has been given to continuous acoustic communication.
Keyword localisation in untranscribed speech using visually grounded speech models
Masked-based localisation gives some of the best reported localisation scores from a VGS model, with an accuracy of 57% when the system knows that a keyword occurs in an utterance and need to predict its location.
Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoring
This paper instead revisits an older approach to word segmentation: bottom-up phone-like unit discovery is performed first, and symbolic word segmentation is then performed on top of the discovered units (without influencing the lower level).
A Temporal Extension of Latent Dirichlet Allocation for Unsupervised Acoustic Unit Discovery
In this paper, we present an extension to LDA that uses a Markov chain to model temporal information.
YFACC: A Yorùbá speech-image dataset for cross-lingual keyword localisation through visual grounding
We collect and release a new single-speaker dataset of audio captions for 6k Flickr images in Yor\`ub\'a -- a real low-resource language spoken in Nigeria.
GAN You Hear Me? Reclaiming Unconditional Speech Synthesis from Diffusion Models
As in the StyleGAN family of image synthesis models, ASGAN maps sampled noise to a disentangled latent vector which is then mapped to a sequence of audio features so that signal aliasing is suppressed at every layer.
Towards visually prompted keyword localisation for zero-resource spoken languages
We formalise this task and call it visually prompted keyword localisation (VPKL): given an image of a keyword, detect and predict where in an utterance the keyword occurs.
TransFusion: Transcribing Speech with Multinomial Diffusion
In this work we aim to see whether the benefits of diffusion models can also be realized for speech recognition.
Mitigating Catastrophic Forgetting for Few-Shot Spoken Word Classification Through Meta-Learning
We consider the problem of few-shot spoken word classification in a setting where a model is incrementally introduced to new word classes.
Visually grounded few-shot word acquisition with fewer shots
Our approach involves using the given word-image example pairs to mine new unsupervised word-image training pairs from large collections of unlabelled speech and images.
Voice Conversion With Just Nearest Neighbors
Any-to-any voice conversion aims to transform source speech into a target voice with just a few examples of the target speaker as a reference.
Ranked #1 on
Voice Conversion
on LibriSpeech test-clean
(using extra training data)
Towards hate speech detection in low-resource languages: Comparing ASR to acoustic word embeddings on Wolof and Swahili
But in an in-the-wild test on Swahili radio broadcasts with actual hate speech keywords, the AWE model (using one minute of template data) is more robust, giving similar performance to an ASR system trained on 30 hours of labelled data.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Visually grounded few-shot word learning in low-resource settings
We propose an approach that can work on natural word-image pairs but with less examples, i. e. fewer shots, and then illustrate how this approach can be applied for multimodal few-shot learning in a real low-resource language, Yor\`ub\'a.
Disentanglement in a GAN for Unconditional Speech Synthesis
We confirm that ASGAN's latent space is disentangled: we demonstrate how simple linear operations in the space can be used to perform several tasks unseen during training.
Leveraging multilingual transfer for unsupervised semantic acoustic word embeddings
Acoustic word embeddings (AWEs) are fixed-dimensional vector representations of speech segments that encode phonetic content so that different realisations of the same word have similar embeddings.
Rhythm Modeling for Voice Conversion
Voice conversion aims to transform source speech into a different target voice.
Voice Conversion for Stuttered Speech, Instruments, Unseen Languages and Textually Described Voices
Nevertheless, this shows that voice conversion models - and kNN-VC in particular - are increasingly applicable in a range of non-standard downstream tasks.
Revisiting speech segmentation and lexicon learning with better features
We revisit a self-supervised method that segments unlabelled speech into word-like segments.
Visually Grounded Speech Models have a Mutual Exclusivity Bias
To simulate prior acoustic and visual knowledge, we experiment with several initialisation strategies using pretrained speech and vision networks.
LiSTra Automatic Speech Translation: English to Lingala Case Study
In recent years there has been great interest in addressing the data scarcity of African languages and providing baseline models for different Natural Language Processing tasks (Orife et al., 2020).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
