Search Results for author:
Found 82 papers, 51 papers with code
SpiRit-LM: Interleaved Spoken and Written Language Model
We introduce SPIRIT-LM, a foundation multimodal language model that freely mixes text and speech.
Getting the most out of your tokenizer for pre-training and domain adaptation
Tokenization is an understudied and often neglected component of modern LLMs.
Masked Audio Generation using a Single Non-Autoregressive Transformer
We introduce MAGNeT, a masked generative sequence modeling method that operates directly over several streams of audio tokens.
CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
The best setup, GPT-4 with chain of thought (CoT), achieves a pass@1 of 75% and 81% on input and output prediction, respectively.
Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models
This paper presents CyberSecEval, a comprehensive benchmark developed to help bolster the cybersecurity of Large Language Models (LLMs) employed as coding assistants.
Generative Spoken Language Model based on continuous word-sized audio tokens
In NLP, text language models based on words or subwords are known to outperform their character-based counterparts.
A Data Source for Reasoning Embodied Agents
In this work, to further pursue these advances, we introduce a new data generator for machine reasoning that integrates with an embodied agent.
Large Language Models for Compiler Optimization
We explore the novel application of Large Language Models to code optimization.
Code Llama: Open Foundation Models for Code
We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks.
Ranked #12 on
Code Generation
on HumanEval
EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis
Recent work has shown that it is possible to resynthesize high-quality speech based, not on text, but on low bitrate discrete units that have been learned in a self-supervised fashion and can therefore capture expressive aspects of speech that are hard to transcribe (prosody, voice styles, non-verbal vocalization).
Simple and Controllable Music Generation
We tackle the task of conditional music generation.
Ranked #4 on
Text-to-Music Generation
on MusicCaps
Textually Pretrained Speech Language Models
In this work, we propose TWIST, a method for training SpeechLMs using a warm-start from a pretrained textual language models.
DINOv2: Learning Robust Visual Features without Supervision
The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision.
Ranked #1 on
Image Classification
on CIFAR-10
Contrastive Distillation Is a Sample-Efficient Self-Supervised Loss Policy for Transfer Learning
Traditional approaches to RL have focused on learning decision policies directly from episodic decisions, while slowly and implicitly learning the semantics of compositional representations needed for generalization.
Leveraging Demonstrations with Latent Space Priors
Starting with a learned joint latent space, we separately train a generative model of demonstration sequences and an accompanying low-level policy.
High Fidelity Neural Audio Compression
We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks.
Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling
This work focuses on improving the robustness of discrete input representations for generative spoken language modeling.
AudioGen: Textually Guided Audio Generation
Finally, we explore the ability of the proposed method to generate audio continuation conditionally and unconditionally.
Ranked #12 on
Audio Generation
on AudioCaps
Code Translation with Compiler Representations
Here we propose to augment code translation with IRs, specifically LLVM IR, with results on the C++, Java, Rust, and Go languages.
Flashlight: Enabling Innovation in Tools for Machine Learning
This is in part due to the difficulties involved in prototyping new computational paradigms with existing frameworks.
Star Temporal Classification: Sequence Classification with Partially Labeled Data
These experiments show that STC can recover most of the performance of supervised baseline when up to 70% of the labels are missing.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Augmenting Convolutional networks with attention-based aggregation
We show how to augment any convolutional network with an attention-based global map to achieve non-local reasoning.
Ranked #38 on
Semantic Segmentation
on ADE20K val
Pseudo-Labeling for Massively Multilingual Speech Recognition
Semi-supervised learning through pseudo-labeling has become a staple of state-of-the-art monolingual speech recognition systems.
Hierarchical Skills for Efficient Exploration
We alleviate the need for prior knowledge by proposing a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner.
ASR4REAL: An extended benchmark for speech models
Popular ASR benchmarks such as Librispeech and Switchboard are limited in the diversity of settings and speakers they represent.
Leveraging Automated Unit Tests for Unsupervised Code Translation
With little to no parallel data available for programming languages, unsupervised methods are well-suited to source code translation.
Word Order Does Not Matter For Speech Recognition
In this paper, we study training of automatic speech recognition system in a weakly supervised setting where the order of words in transcript labels of the audio training data is not known.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
XCiT: Cross-Covariance Image Transformers
We propose a "transposed" version of self-attention that operates across feature channels rather than tokens, where the interactions are based on the cross-covariance matrix between keys and queries.
Ranked #55 on
Instance Segmentation
on COCO minival
CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings
Absolute or relative positional embeddings are the most popular ways to feed Transformer models with positional information.
ResMLP: Feedforward networks for image classification with data-efficient training
We present ResMLP, an architecture built entirely upon multi-layer perceptrons for image classification.
Ranked #1 on
Image Classification
on Certificate Verification
MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding
We also investigate the utility of our model as an object detector on a given label set when fine-tuned in a few-shot setting.
Ranked #1 on
Visual Question Answering (VQA)
on CLEVR-Humans
Generalized Referring Expression Comprehension
Phrase Grounding
+9
Gradient Matching for Domain Generalization
We perform experiments on both the Wilds benchmark, which captures distribution shift in the real world, as well as datasets in DomainBed benchmark that focuses more on synthetic-to-real transfer.
Differentiable Model Compression via Pseudo Quantization Noise
DiffQ is differentiable both with respect to the unquantized weights and the number of bits used.
Ranked #29 on
Language Modelling
on WikiText-103
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training
On a large-scale competitive setup, we show that pre-training on unlabeled in-domain data reduces the gap between models trained on in-domain and out-of-domain labeled data by 66%-73%.
Going deeper with Image Transformers
In particular, we investigate the interplay of architecture and optimization of such dedicated transformers.
Ranked #5 on
Image Classification
on CIFAR-10
(using extra training data)
MDETR - Modulated Detection for End-to-End Multi-Modal Understanding
We also investigate the utility of our model as an object detector on a given label set when fine-tuned in a few-shot setting.
Ranked #2 on
Referring Expression Comprehension
on Talk2Car
(using extra training data)
ROMUL: Scale Adaptative Population Based Training
In most pragmatic settings, data augmentation and regularization are essential, and require hyperparameter search.
MLS: A Large-Scale Multilingual Dataset for Speech Research
This paper introduces Multilingual LibriSpeech (MLS) dataset, a large multilingual corpus suitable for speech research.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Joint Masked CPC and CTC Training for ASR
Self-supervised learning (SSL) has shown promise in learning representations of audio that are useful for automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
SlimIPL: Language-Model-Free Iterative Pseudo-Labeling
We improve upon the IPL algorithm: as the model learns, we propose to iteratively re-generate transcriptions with hard labels (the most probable tokens), that is, without a language model.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Self-training and Pre-training are Complementary for Speech Recognition
Self-training and unsupervised pre-training have emerged as effective approaches to improve speech recognition systems using unlabeled data.
Ranked #1 on
Speech Recognition
on LibriSpeech train-clean-100 test-other
(using extra training data)
Rethinking Evaluation in ASR: Are Our Models Robust Enough?
Finally, we show that training a single acoustic model on the most widely-used datasets - combined - reaches competitive performance on both research and real-world benchmarks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Population Based Training for Data Augmentation and Regularization in Speech Recognition
It compares favorably to a baseline that does not change those hyperparameters over the course of training, with an 8% relative WER improvement.
Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters
We study training a single acoustic model for multiple languages with the aim of improving automatic speech recognition (ASR) performance on low-resource languages, and over-all simplifying deployment of ASR systems that support diverse languages.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Data Augmenting Contrastive Learning of Speech Representations in the Time Domain
Contrastive Predictive Coding (CPC), based on predicting future segments of speech based on past segments is emerging as a powerful algorithm for representation learning of speech signal.
Real Time Speech Enhancement in the Waveform Domain
The proposed model matches state-of-the-art performance of both causal and non causal methods while working directly on the raw waveform.
End-to-End Object Detection with Transformers
We present a new method that views object detection as a direct set prediction problem.
Ranked #21 on
Panoptic Segmentation
on COCO minival
Iterative Pseudo-Labeling for Speech Recognition
In particular, IPL fine-tunes an existing model at each iteration using both labeled data and a subset of unlabeled data.
Ranked #11 on
Speech Recognition
on LibriSpeech test-other
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Contextualizing ASR Lattice Rescoring with Hybrid Pointer Network Language Model
Videos uploaded on social media are often accompanied with textual descriptions.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Semi-Supervised Speech Recognition via Local Prior Matching
For sequence transduction tasks like speech recognition, a strong structured prior model encodes rich information about the target space, implicitly ruling out invalid sequences by assigning them low probability.
Ranked #45 on
Speech Recognition
on LibriSpeech test-other
Polygames: Improved Zero Learning
Since DeepMind's AlphaZero, Zero learning quickly became the state-of-the-art method for many board games.
Scaling Up Online Speech Recognition Using ConvNets
We design an online end-to-end speech recognition system based on Time-Depth Separable (TDS) convolutions and Connectionist Temporal Classification (CTC).
Libri-Light: A Benchmark for ASR with Limited or No Supervision
Additionally, we provide baseline systems and evaluation metrics working under three settings: (1) the zero resource/unsupervised setting (ABX), (2) the semi-supervised setting (PER, CER) and (3) the distant supervision setting (WER).
Ranked #1 on
Speech Recognition
on Libri-Light test-other
(ABX-within metric)
End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
We study pseudo-labeling for the semi-supervised training of ResNet, Time-Depth Separable ConvNets, and Transformers for speech recognition, with either CTC or Seq2Seq loss functions.
Ranked #16 on
Speech Recognition
on LibriSpeech test-other
(using extra training data)
Deja-vu: Double Feature Presentation and Iterated Loss in Deep Transformer Networks
As our motivation is to allow acoustic models to re-examine their input features in light of partial hypotheses we introduce intermediate model heads and loss function.
A Structured Prediction Approach for Generalization in Cooperative Multi-Agent Reinforcement Learning
While centralized reinforcement learning methods can optimally solve small MAC instances, they do not scale to large problems and they fail to generalize to scenarios different from those seen during training.
Multi-agent Reinforcement Learning
reinforcement-learning
+4
Self-Supervised Speech Recognition via Local Prior Matching
We propose local prior matching (LPM), a self-supervised objective for speech recognition.
Why Build an Assistant in Minecraft?
In this document we describe a rationale for a research program aimed at building an open "assistant" in the game Minecraft, in order to make progress on the problems of natural language understanding and learning from dialogue.
Growing Action Spaces
In complex tasks, such as those with large combinatorial action spaces, random exploration may be too inefficient to achieve meaningful learning progress.
Word-level Speech Recognition with a Letter to Word Encoder
We propose a direct-to-word sequence model which uses a word network to learn word embeddings from letters.
Who Needs Words? Lexicon-Free Speech Recognition
Lexicon-free speech recognition naturally deals with the problem of out-of-vocabulary (OOV) words.
A Fully Differentiable Beam Search Decoder
We demonstrate our approach scales by applying it to speech recognition, jointly training acoustic and word-level language models.
wav2letter++: The Fastest Open-source Speech Recognition System
This paper introduces wav2letter++, the fastest open-source deep learning speech recognition framework.
Fully Convolutional Speech Recognition
In this paper we present an alternative approach based solely on convolutional neural networks, leveraging recent advances in acoustic models from the raw waveform and language modeling.
Ranked #3 on
Speech Recognition
on WSJ eval93
To Reverse the Gradient or Not: An Empirical Comparison of Adversarial and Multi-task Learning in Speech Recognition
In multi-task learning, the goal is speaker prediction; we expect a performance improvement with this joint training if the two tasks of speech recognition and speaker recognition share a common set of underlying features.
Forward Modeling for Partial Observation Strategy Games - A StarCraft Defogger
We formulate the problem of defogging as state estimation and future state prediction from previous, partial observations in the context of real-time strategy games.
High-Level Strategy Selection under Partial Observability in StarCraft: Brood War
We consider the problem of high-level strategy selection in the adversarial setting of real-time strategy games from a reinforcement learning perspective, where taking an action corresponds to switching to the respective strategy.
End-to-End Speech Recognition From the Raw Waveform
In this paper, we study end-to-end systems trained directly from the raw waveform, building on two alternatives for trainable replacements of mel-filterbanks that use a convolutional architecture.
Value Propagation Networks
We present Value Propagation (VProp), a set of parameter-efficient differentiable planning modules built on Value Iteration which can successfully be trained using reinforcement learning to solve unseen tasks, has the capability to generalize to larger map sizes, and can learn to navigate in dynamic environments.
Gated ConvNets for Letter-Based ASR
In this paper we introduce a new speech recognition system, leveraging a simple letter-based ConvNet acoustic model.
Letter-Based Speech Recognition with Gated ConvNets
In the recent literature, "end-to-end" speech systems often refer to letter-based acoustic models trained in a sequence-to-sequence manner, either via a recurrent model or via a structured output learning approach (such as CTC).
Ranked #46 on
Speech Recognition
on LibriSpeech test-clean
Learning Filterbanks from Raw Speech for Phone Recognition
We train a bank of complex filters that operates on the raw waveform and is fed into a convolutional neural network for end-to-end phone recognition.
STARDATA: A StarCraft AI Research Dataset
We provide full game state data along with the original replays that can be viewed in StarCraft.
Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
When Bob is deployed on an RL task within the environment, this unsupervised training reduces the number of supervised episodes needed to learn, and in some cases converges to a higher reward.
TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games
We present TorchCraft, a library that enables deep learning research on Real-Time Strategy (RTS) games such as StarCraft: Brood War, by making it easier to control these games from a machine learning framework, here Torch.
Wav2Letter: an End-to-End ConvNet-based Speech Recognition System
This paper presents a simple end-to-end model for speech recognition, combining a convolutional network based acoustic model and a graph decoding.
Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks
We consider scenarios from the real-time strategy game StarCraft as new benchmarks for reinforcement learning algorithms.
MazeBase: A Sandbox for Learning from Games
This paper introduces MazeBase: an environment for simple 2D games, designed as a sandbox for machine learning approaches to reasoning and planning.
Weakly Supervised Multi-Embeddings Learning of Acoustic Models
We trained a Siamese network with multi-task same/different information on a speech dataset, and found that it was possible to share a network for both tasks without a loss in performance.
A Dataset for StarCraft AI \& an Example of Armies Clustering
We evaluated this clustering method by predicting the outcomes of battles based on armies compositions' mixtures components
