Search Results for author:
Found 56 papers, 30 papers with code
Towards Robust and Efficient Continual Language Learning
As the application space of language models continues to evolve, a natural question to ask is how we can quickly adapt models to new tasks.
Towards Compute-Optimal Transfer Learning
The field of transfer learning is undergoing a significant shift with the introduction of large pretrained models which have demonstrated strong adaptability to a variety of downstream tasks.
NEVIS'22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision Research
A shared goal of several machine learning communities like continual learning, meta-learning and transfer learning, is to design algorithms and models that efficiently and robustly adapt to unseen tasks.
Multi-step Planning for Automated Hyperparameter Optimization with OptFormer
As machine learning permeates more industries and models become more expensive and time consuming to train, the need for efficient automated hyperparameter optimization (HPO) has never been more pressing.
Towards Learning Universal Hyperparameter Optimizers with Transformers
Meta-learning hyperparameter optimization (HPO) algorithms from prior experiments is a promising approach to improve optimization efficiency over objective functions from a similar distribution.
On Anytime Learning at Macroscale
Practitioners have then to decide how to allocate their computational budget in order to obtain the best performance at any point in time.
The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation
One of the biggest challenges hindering progress in low-resource and multilingual machine translation is the lack of good evaluation benchmarks.
Efficient Continual Learning with Modular Networks and Task-Driven Priors
Finally, we introduce a new modular architecture, whose modules represent atomic skills that can be composed to perform a certain task.
Few-shot Sequence Learning with Transformers
In the simplest setting, we append a token to an input sequence which represents the particular task to be undertaken, and show that the embedding of this token can be optimized on the fly given few labeled examples.
Multi-scale Transformer Language Models
We investigate multi-scale transformer language models that learn representations of text at multiple scales, and present three different architectures that have an inductive bias to handle the hierarchical nature of language.
Residual Energy-Based Models for Text Generation
In this work, we investigate un-normalized energy-based models (EBMs) which operate not at the token but at the sequence level.
Residual Energy-Based Models for Text
Current large-scale auto-regressive language models display impressive fluency and can generate convincing text.
Facebook AI's WAT19 Myanmar-English Translation Task Submission
This paper describes Facebook AI's submission to the WAT 2019 Myanmar-English translation task.
Revisiting Self-Training for Neural Sequence Generation
In this work, we first empirically show that self-training is able to decently improve the supervised baseline on neural sequence generation tasks.
The Source-Target Domain Mismatch Problem in Machine Translation
While we live in an increasingly interconnected world, different places still exhibit strikingly different cultures and many events we experience in our every day life pertain only to the specific place we live in.
On The Evaluation of Machine Translation Systems Trained With Back-Translation
Back-translation is a widely used data augmentation technique which leverages target monolingual data.
Large Memory Layers with Product Keys
In our experiments we consider a dataset with up to 30 billion words, and we plug our memory layer in a state-of-the-art transformer-based architecture.
Real or Fake? Learning to Discriminate Machine from Human Generated Text
Energy-based models (EBMs), a. k. a.
Task-Driven Modular Networks for Zero-Shot Compositional Learning
When extending the evaluation to the generalized setting which accounts also for pairs seen during training, we discover that naive baseline methods perform similarly or better than current approaches.
Multiple-Attribute Text Rewriting
The dominant approach to unsupervised "style transfer" in text is based on the idea of learning a latent representation, which is independent of the attributes specifying its "style".
On Tiny Episodic Memories in Continual Learning
But for a successful knowledge transfer, the learner needs to remember how to perform previous tasks.
Ranked #7 on
Class Incremental Learning
on cifar100
Mixture Models for Diverse Machine Translation: Tricks of the Trade
Facebook AI Research Sequence-to-Sequence Toolkit written in Python.
The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English
For machine translation, a vast majority of language pairs in the world are considered low-resource because they have little parallel data available.
Efficient Lifelong Learning with A-GEM
In lifelong learning, the learner is presented with a sequence of tasks, incrementally building a data-driven prior which may be leveraged to speed up learning of a new task.
Ranked #6 on
Continual Learning
on ASC (19 tasks)
Multiple-Attribute Text Style Transfer
The dominant approach to unsupervised "style transfer" in text is based on the idea of learning a latent representation, which is independent of the attributes specifying its "style".
GenEval: A Benchmark Suite for Evaluating Generative Models
In this work, we aim at addressing this problem by introducing a new benchmark evaluation suite, dubbed \textit{GenEval}.
Lightweight Adaptive Mixture of Neural and N-gram Language Models
It is often the case that the best performing language model is an ensemble of a neural language model with n-grams.
Phrase-Based & Neural Unsupervised Machine Translation
Machine translation systems achieve near human-level performance on some languages, yet their effectiveness strongly relies on the availability of large amounts of parallel sentences, which hinders their applicability to the majority of language pairs.
Ranked #2 on
Machine Translation
on WMT2016 English-Russian
Analyzing Uncertainty in Neural Machine Translation
We propose tools and metrics to assess how uncertainty in the data is captured by the model distribution and how it affects search strategies that generate translations.
Fader Networks:Manipulating Images by Sliding Attributes
This paper introduces a new encoder-decoder architecture that is trained to reconstruct images by disentangling the salient information of the image and the values of attributes directly in the latent space.
Classical Structured Prediction Losses for Sequence to Sequence Learning
There has been much recent work on training neural attention models at the sequence-level using either reinforcement learning-style methods or by optimizing the beam.
Ranked #4 on
Machine Translation
on IWSLT2015 German-English
Unsupervised Machine Translation Using Monolingual Corpora Only
By learning to reconstruct in both languages from this shared feature space, the model effectively learns to translate without using any labeled data.
Ranked #6 on
Machine Translation
on WMT2016 German-English
Word Translation Without Parallel Data
We finally describe experiments on the English-Esperanto low-resource language pair, on which there only exists a limited amount of parallel data, to show the potential impact of our method in fully unsupervised machine translation.
Ranked #2 on
Word Alignment
on en-es
Gradient Episodic Memory for Continual Learning
One major obstacle towards AI is the poor ability of models to solve new problems quicker, and without forgetting previously acquired knowledge.
Fader Networks: Manipulating Images by Sliding Attributes
This paper introduces a new encoder-decoder architecture that is trained to reconstruct images by disentangling the salient information of the image and the values of attributes directly in the latent space.
Hard Mixtures of Experts for Large Scale Weakly Supervised Vision
In this work we show that a simple hard mixture of experts model can be efficiently trained to good effect on large scale hashtag (multilabel) prediction tasks.
Training Language Models Using Target-Propagation
While Truncated Back-Propagation through Time (BPTT) is the most popular approach to training Recurrent Neural Networks (RNNs), it suffers from being inherently sequential (making parallelization difficult) and from truncating gradient flow between distant time-steps.
Transformation-Based Models of Video Sequences
In this work we propose a simple unsupervised approach for next frame prediction in video.
Learning through Dialogue Interactions by Asking Questions
A good dialogue agent should have the ability to interact with users by both responding to questions and by asking questions, and importantly to learn from both types of interaction.
Dialogue Learning With Human-In-The-Loop
An important aspect of developing conversational agents is to give a bot the ability to improve through communicating with humans and to learn from the mistakes that it makes.
Sequence Level Training with Recurrent Neural Networks
Many natural language processing applications use language models to generate text.
Ranked #14 on
Machine Translation
on IWSLT2015 German-English
Convolutional networks and learning invariant to homogeneous multiplicative scalings
The conventional classification schemes -- notably multinomial logistic regression -- used in conjunction with convolutional networks (convnets) are classical in statistics, designed without consideration for the usual coupling with convnets, stochastic gradient descent, and backpropagation.
Learning Longer Memory in Recurrent Neural Networks
In this paper, we show that learning longer term patterns in real data, such as in natural language, is perfectly possible using gradient descent.
Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews
Sentiment analysis is a common task in natural language processing that aims to detect polarity of a text document (typically a consumer review).
Web-Scale Training for Face Identification
Scaling machine learning methods to very large datasets has attracted considerable attention in recent years, thanks to easy access to ubiquitous sensing and data from the web.
On Learning Where To Look
Current automatic vision systems face two major challenges: scalability and extreme variability of appearance.
Multi-GPU Training of ConvNets
In this work we evaluate different approaches to parallelize computation of convolutional neural networks across several GPUs.
Learning Factored Representations in a Deep Mixture of Experts
In addition, we see that the different combinations are in use when the model is applied to a dataset of speech monophones.
DeViSE: A Deep Visual-Semantic Embedding Model
Modern visual recognition systems are often limited in their ability to scale to large numbers of object categories.
Ranked #10 on
Zero-Shot Action Recognition
on Kinetics
PANDA: Pose Aligned Networks for Deep Attribute Modeling
We propose a method for inferring human attributes (such as gender, hair style, clothes style, expression, action) from images of people under large variation of viewpoint, pose, appearance, articulation and occlusion.
Predicting Parameters in Deep Learning
We demonstrate that there is significant redundancy in the parameterization of several deep learning models.
Large Scale Distributed Deep Networks
Recent work in unsupervised feature learning and deep learning has shown that being able to train large models can dramatically improve performance.
Building high-level features using large scale unsupervised learning
For example, is it possible to learn a face detector using only unlabeled images?
Generating more realistic images using gated MRF's
Probabilistic models of natural images are usually evaluated by measuring performance on rather indirect tasks, such as denoising and inpainting.
Phone Recognition with the Mean-Covariance Restricted Boltzmann Machine
Straightforward application of Deep Belief Nets (DBNs) to acoustic modeling produces a rich distributed representation of speech data that is useful for recognition and yields impressive results on the speaker-independent TIMIT phone recognition task.
Sparse Feature Learning for Deep Belief Networks
Unsupervised learning algorithms aim to discover the structure hidden in the data, and to learn representations that are more suitable as input to a supervised machine than the raw input.
