Search Results for author:
Found 91 papers, 39 papers with code
A comparison study on patient-psychologist voice diarization
Conversations between a clinician and a patient, in natural conditions, are valuable sources of information for medical follow-up.
Intuitive physics understanding emerges from self-supervised pretraining on natural videos
We investigate the emergence of intuitive physics understanding in general-purpose deep neural network models trained to predict masked regions in natural videos.
Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approach
Modeling directly from speech opens up the path to more natural and expressive systems.
Simulating Articulatory Trajectories with Phonological Feature Interpolation
Two phonological feature sets, based respectively on generative and articulatory phonology, are used to encode a phonetic target sequence.
Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learning
While Reinforcement Learning (RL) has been proven essential for tuning large language models (LLMs), it can lead to reward over-optimization (ROO).
Language Evolution with Deep Learning
Computational modeling plays an essential role in the study of language emergence.
Spirit LM: Interleaved Spoken and Written Language Model
Our model is based on a 7B pretrained text language model that we extend to the speech modality by continuously training it on text and speech units.
Ranked #1 on
Language Modelling
on SALMon
(using extra training data)
EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Models
We introduce EmphAssess, a prosodic benchmark designed to evaluate the capability of speech-to-speech models to encode and reproduce prosodic emphasis.
WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Models
We propose WorldSense, a benchmark designed to assess the extent to which LLMs are consistently able to sustain tacit world models, by testing how they draw simple inferences from descriptions of simple arrangements of entities.
XLS-R fine-tuning on noisy word boundaries for unsupervised speech segmentation into words
Due to the absence of explicit word boundaries in the speech stream, the task of segmenting spoken sentences into word units without text supervision is particularly challenging.
Generative Spoken Language Model based on continuous word-sized audio tokens
In NLP, text language models based on words or subwords are known to outperform their character-based counterparts.
Low-Resource Self-Supervised Learning with SSL-Enhanced TTS
Self-supervised learning (SSL) techniques have achieved remarkable results in various speech processing tasks.
EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis
Recent work has shown that it is possible to resynthesize high-quality speech based, not on text, but on low bitrate discrete units that have been learned in a self-supervised fashion and can therefore capture expressive aspects of speech that are hard to transcribe (prosody, voice styles, non-verbal vocalization).
BabySLM: language-acquisition-friendly benchmark of self-supervised spoken language models
Self-supervised techniques for learning speech representations have been shown to develop linguistic competence from exposure to speech without the need for human labels.
Textually Pretrained Speech Language Models
In this work, we propose TWIST, a method for training SpeechLMs using a warm-start from a pretrained textual language models.
Ranked #4 on
Language Modelling
on SALMon
(using extra training data)
ProsAudit, a prosodic benchmark for self-supervised speech models
In the lexical task, the model needs to correctly distinguish between pauses inserted between words and within words.
Introducing topography in convolutional neural networks
Thus, in this work, inspired by the neuroscience literature, we proposed a new topographic inductive bias in Convolutional Neural Networks (CNNs).
Evaluating context-invariance in unsupervised speech representations
Unsupervised speech representations have taken off, with benchmarks (SUPERB, ZeroSpeech) demonstrating major progress on semi-supervised speech recognition, speech synthesis, and speech-only language modelling.
Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challenge
Recent progress in self-supervised or unsupervised machine learning has opened the possibility of building a full speech processing system from raw audio without using any textual representations or expert labels such as phonemes, dictionaries or parse trees.
Brouhaha: multi-task training for voice activity detection, speech-to-noise ratio, and C50 room acoustics estimation
Most automatic speech processing systems register degraded performance when applied to noisy or reverberant speech.
Are word boundaries useful for unsupervised language learning?
However, word boundary information may be absent or unreliable in the case of speech input (word boundaries are not marked explicitly in the speech stream).
Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling
This work focuses on improving the robustness of discrete input representations for generative spoken language modeling.
Emergent Communication: Generalization and Overfitting in Lewis Games
Lewis signaling games are a class of simple communication games for simulating the emergence of language.
STOP: A dataset for Spoken Task Oriented Semantic Parsing
Furthermore, in addition to the human-recorded audio, we are releasing a TTS-generated version to benchmark the performance for low-resource domain adaptation of end-to-end SLU systems.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Is the Language Familiarity Effect gradual? A computational modelling approach
According to the Language Familiarity Effect (LFE), people are better at discriminating between speakers of their native language.
DP-Parse: Finding Word Boundaries from Raw Speech with an Instance Lexicon
Finding word boundaries in continuous speech is challenging as there is little or no equivalent of a 'space' delimiter between words.
Speech Sequence Embeddings using Nearest Neighbors Contrastive Learning
We introduce a simple neural encoder architecture that can be trained using an unsupervised contrastive learning objective which gets its positive samples from data-augmented k-Nearest Neighbors search.
Generative Spoken Dialogue Language Modeling
We introduce dGSLM, the first "textless" model able to generate audio samples of naturalistic spoken dialogues.
Probing phoneme, language and speaker information in unsupervised speech representations
Language information, however, is very salient in the bilingual model only, suggesting CPC models learn to discriminate languages when trained on multiple languages.
Are discrete units necessary for Spoken Language Modeling?
The approach relies first on transforming the audio into a sequence of discrete units (or pseudo-text) and then training a language model directly on such pseudo-text.
textless-lib: a Library for Textless Spoken Language Processing
Textless spoken language processing research aims to extend the applicability of standard NLP toolset onto spoken language and languages with few or no textual resources.
Towards Interactive Language Modeling
As a first contribution we present a road map in which we detail the steps that need to be taken towards interactive language modeling.
Shennong: a Python toolbox for audio speech features extraction
After describing the Shennong software architecture, its core components and implemented algorithms, this paper illustrates its use on three applications: a comparison of speech features performances on a phones discrimination task, an analysis of a Vocal Tract Length Normalization model as a function of the speech duration used for training and a comparison of pitch estimation algorithms under various noise conditions.
Textless Speech Emotion Conversion using Decomposed & Discrete Representations
We decompose speech into discrete and disentangled learned representations, consisting of content units, F0, speaker, and emotion.
Textless Speech Emotion Conversion using Discrete and Decomposed Representations
We use a decomposition of the speech signal into discrete learned representations, consisting of phonetic-content units, prosodic features, speaker, and emotion.
Text-Free Prosody-Aware Generative Spoken Language Modeling
Generative Spoken Language Modeling (GSLM) \cite{Lakhotia2021} is the only prior work addressing the generative aspects of speech pre-training, which replaces text with discovered phone-like units for language modeling and shows the ability to generate meaningful novel sentences.
Ranked #8 on
Language Modelling
on SALMon
(using extra training data)
ZR-2021VG: Zero-Resource Speech Challenge, Visually-Grounded Language Modelling track, 2021 edition
We present the visually-grounded language modelling track that was introduced in the Zero-Resource Speech challenge, 2021 edition, 2nd round.
The Zero Resource Speech Challenge 2021: Spoken language modelling
We present the Zero Resource Speech Challenge 2021, which asks participants to learn a language model directly from audio, without any text or labels.
Speech Resynthesis from Discrete Disentangled Self-Supervised Representations
We propose using self-supervised discrete representations for the task of speech resynthesis.
Learning spectro-temporal representations of complex sounds with parameterized neural networks
We found out that models based on Learnable STRFs are on par for all tasks with different toplines, and obtain the best performance for Speech Activity Detection.
Generative Spoken Language Modeling from Raw Audio
We introduce Generative Spoken Language Modeling, the task of learning the acoustic and linguistic characteristics of a language from raw audio (no text, no labels), and a set of metrics to automatically evaluate the learned representations at acoustic and linguistic levels for both encoding and generation.
Ranked #1 on
Resynthesis
on LibriSpeech
VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation
We introduce VoxPopuli, a large-scale multilingual corpus providing 100K hours of unlabelled speech data in 23 languages.
Ranked #3 on
Speech Recognition
on Common Voice French
(using extra training data)
The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling
We introduce a new unsupervised task, spoken language modeling: the learning of linguistic representations from raw audio signals without any labels, along with the Zero Resource Speech Benchmark 2021: a suite of 4 black-box, zero-shot metrics probing for the quality of the learned models at 4 linguistic levels: phonetics, lexicon, syntax and semantics.
``LazImpa'': Lazy and Impatient neural agents learn to communicate efficiently
Previous work has shown that artificial neural agents naturally develop surprisingly non-efficient codes.
Comparison of Speaker Role Recognition and Speaker Enrollment Protocol for conversational Clinical Interviews
Here, we proposed a split of the data that allows conducting a comparative evaluation of speaker role recognition and speaker enrollment methods to solve this task.
The Zero Resource Speech Challenge 2020: Discovering discrete subword and word units
We present the Zero Resource Speech Challenge 2020, which aims at learning speech representations from raw audio signals without any labels.
Analogies minus analogy test: measuring regularities in word embeddings
Vector space models of words have long been claimed to capture linguistic regularities as simple vector translations, but problems have been raised with this claim.
"LazImpa": Lazy and Impatient neural agents learn to communicate efficiently
Previous work has shown that artificial neural agents naturally develop surprisingly non-efficient codes.
Evaluating the reliability of acoustic speech embeddings
However, there is currently no clear methodology to compare or optimise the quality of these embeddings in a task-neutral way.
Data Augmenting Contrastive Learning of Speech Representations in the Time Domain
Contrastive Predictive Coding (CPC), based on predicting future segments of speech based on past segments is emerging as a powerful algorithm for representation learning of speech signal.
Vocal markers from sustained phonation in Huntington's Disease
According to our regression results, Phonatory features are suitable for the predictions of clinical performance in Huntington's Disease.
An open-source voice type classifier for child-centered daylong recordings
Spontaneous conversations in real-world settings such as those found in child-centered recordings have been shown to be amongst the most challenging audio files to process.
Occlusion resistant learning of intuitive physics from videos
In this work we propose a probabilistic formulation of learning intuitive physics in 3D scenes with significant inter-object occlusions.
Compositionality and Generalization in Emergent Languages
Third, while compositionality is not necessary for generalization, it provides an advantage in terms of language transmission: The more compositional a language is, the more easily it will be picked up by new learners, even when the latter differ in architecture from the original agents.
Seshat: A tool for managing and verifying annotation campaigns of audio data
We introduce Seshat, a new, simple and open-source software to efficiently manage annotations of speech corpora.
Identification of primary and collateral tracks in stuttered speech
Here, we introduce a new evaluation framework for disfluency detection inspired by the clinical and NLP perspective together with the theory of performance from \cite{clark1996using} which distinguishes between primary and collateral tracks.
Unsupervised pretraining transfers well across languages
Cross-lingual and multi-lingual training of Automatic Speech Recognition (ASR) has been extensively investigated in the supervised setting.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Libri-Light: A Benchmark for ASR with Limited or No Supervision
Additionally, we provide baseline systems and evaluation metrics working under three settings: (1) the zero resource/unsupervised setting (ABX), (2) the semi-supervised setting (PER, CER) and (3) the distant supervision setting (WER).
Ranked #1 on
Speech Recognition
on Libri-Light test-other
(ABX-within metric)
Speaker detection in the wild: Lessons learned from JSALT 2019
This paper presents the problems and solutions addressed at the JSALT workshop when using a single microphone for speaker detection in adverse scenarios.
Audio and Speech Processing Sound
Neural language modeling of free word order argument structure
Neural language models trained with a predictive or masked objective have proven successful at capturing short and long distance syntactic dependencies.
Word-order biases in deep-agent emergent communication
We train models to communicate about paths in a simple gridworld, using miniature languages that reflect or violate various natural language trends, such as the tendency to avoid redundancy or to minimize long-distance dependencies.
Anti-efficient encoding in emergent communication
Despite renewed interest in emergent language simulations with neural networks, little is known about the basic properties of the induced code, and how they compare to human language.
The Zero Resource Speech Challenge 2019: TTS without T
We present the Zero Resource Speech Challenge 2019, which proposes to build a speech synthesizer without any text or phonetic labels: hence, TTS without T (text-to-speech without text).
End-to-End Speech Recognition From the Raw Waveform
In this paper, we study end-to-end systems trained directly from the raw waveform, building on two alternatives for trainable replacements of mel-filterbanks that use a convolutional architecture.
Sampling strategies in Siamese Networks for unsupervised speech representation learning
We apply these results to pairs of words discovered using an unsupervised algorithm and show an improvement on state-of-the-art in unsupervised representation learning using siamese networks.
IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning
In order to reach human performance on complexvisual tasks, artificial systems need to incorporate a sig-nificant amount of understanding of the world in termsof macroscopic objects, movements, forces, etc.
Bayesian Models for Unit Discovery on a Very Low Resource Language
Developing speech technologies for low-resource languages has become a very active research field over the last decade.
Linguistic unit discovery from multi-modal inputs in unwritten languages: Summary of the "Speaking Rosetta" JSALT 2017 Workshop
We summarize the accomplishments of a multi-disciplinary workshop exploring the computational and scientific issues surrounding the discovery of linguistic units (subwords and words) in a language without orthography.
Are words easier to learn from infant- than adult-directed speech? A quantitative corpus-based investigation
At the acoustic level we show that, as has been documented before for phonemes, the realizations of words are more variable and less discriminable in IDS than in ADS.
The Zero Resource Speech Challenge 2017
We describe a new challenge aimed at discovering subword and word units from raw speech.
Learning Filterbanks from Raw Speech for Phone Recognition
We train a bank of complex filters that operates on the raw waveform and is fed into a convolutional neural network for end-to-end phone recognition.
The Role of Prosody and Speech Register in Word Segmentation: A Computational Modelling Perspective
This study explores the role of speech register and prosody for the task of word segmentation.
Learning weakly supervised multimodal phoneme embeddings
Recent works have explored deep architectures for learning multimodal speech representation (e. g. audio and images, articulation and audio) in a supervised way.
Comparing Character-level Neural Language Models Using a Lexical Decision Task
What is the information captured by neural network models of language?
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
The success of long short-term memory (LSTM) neural networks in language processing is typically attributed to their ability to capture long-distance statistical regularities.
Blind phoneme segmentation with temporal prediction errors
Phonemic segmentation of speech is a critical step of speech recognition systems.
Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner
The project of 'reverse engineering' language development, i. e., of building an effective system that mimics infant's achievements appears therefore to be within reach.
Weakly Supervised Multi-Embeddings Learning of Acoustic Models
We trained a Siamese network with multi-task same/different information on a speech dataset, and found that it was possible to share a network for both tasks without a loss in performance.
Bridging the gap between speech technology and natural language processing: an evaluation toolbox for term discovery systems
The unsupervised discovery of linguistic terms from either continuous phoneme transcriptions or from raw speech has seen an increasing interest in the past years both from a theoretical and a practical standpoint.
