Search Results for author:
Found 60 papers, 39 papers with code
Unsupervised Hyper-alignment for Multilingual Word Embeddings
This paper extends this line of work to the problem of aligning multiple languages to a common space.
Neutral residues: revisiting adapters for model extension
We address the problem of extending a pretrained large language model to a new domain that was not seen at training time, like adding a language for which the original model has seen no or little training data.
Moshi: a speech-text foundation model for real-time dialogue
Our resulting model is the first real-time full-duplex spoken large language model, with a theoretical latency of 160ms, 200ms in practice, and is available at https://github. com/kyutai-labs/moshi.
Time Sensitive Knowledge Editing through Efficient Finetuning
It is thus essential to design effective methods to both update obsolete knowledge and induce new knowledge into LLMs.
PaSS: Parallel Speculative Sampling
As an alternative, we propose to use parallel decoding as a way to draft multiple tokens from a single model with no computational cost, nor the need for a second model.
LLaMA: Open and Efficient Foundation Language Models
We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters.
Ranked #3 on
Few-Shot Learning
on MedConceptsQA
Augmented Language Models: a Survey
This survey reviews works in which language models (LMs) are augmented with reasoning skills and the ability to use tools.
EditEval: An Instruction-Based Benchmark for Text Improvements
Evaluation of text generation to date has primarily focused on content created sequentially, rather than improvements on a piece of text.
PEER: A Collaborative Language Model
Textual content is often the output of a collaborative writing process: We start with an initial draft, ask for suggestions, and repeatedly make changes.
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Retrieval augmented models are known to excel at knowledge intensive tasks without the need for as many parameters, but it is unclear whether they work in few-shot settings.
Ranked #1 on
Question Answering
on Natural Questions
Improving Wikipedia Verifiability with AI
Hence, maintaining and improving the quality of Wikipedia references is an important challenge and there is a pressing need for better tools to assist humans in this effort.
Flashlight: Enabling Innovation in Tools for Machine Learning
This is in part due to the difficulties involved in prototyping new computational paradigms with existing frameworks.
Are Large-scale Datasets Necessary for Self-Supervised Pre-training?
Our study shows that denoising autoencoders, such as BEiT or a variant that we introduce in this paper, are more robust to the type and size of the pre-training data than popular self-supervised methods trained by comparing image embeddings. We obtain competitive performance compared to ImageNet pre-training on a variety of classification datasets, from different domains.
The Web Is Your Oyster -- Knowledge-Intensive NLP against a Very Large Web Corpus
In order to address increasing demands of real-world applications, the research for knowledge-intensive NLP (KI-NLP) should advance by capturing the challenges of a truly open-domain environment: web-scale knowledge, lack of structure, inconsistent quality and noise.
Unsupervised Dense Information Retrieval with Contrastive Learning
In this work, we explore the limits of contrastive learning as a way to train unsupervised dense retrievers and show that it leads to strong performance in various retrieval settings.
Contrastive Pre-training for Zero-Shot Information Retrieval
By contrast, in many other NLP tasks, conventional self-supervised pre-training based on masking leads to strong generalization with small number of training examples.
ResMLP: Feedforward networks for image classification with data-efficient training
We present ResMLP, an architecture built entirely upon multi-layer perceptrons for image classification.
Ranked #1 on
Image Classification
on Certificate Verification
NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
We review the EfficientQA competition from NeurIPS 2020.
A Memory Efficient Baseline for Open Domain Question Answering
Recently, retrieval systems based on dense representations have led to important improvements in open-domain question answering, and related tasks.
Distilling Knowledge from Reader to Retriever for Question Answering
A challenge of using such methods is to obtain supervised data to train the retriever model, corresponding to pairs of query and support documents.
Ranked #9 on
Question Answering
on NarrativeQA
Beyond English-Centric Multilingual Machine Translation
Existing work in translation demonstrated the potential of massively multilingual machine translation by training a single model able to translate between any pair of languages.
Self-training Improves Pre-training for Natural Language Understanding
Unsupervised pre-training has led to much recent progress in natural language understanding.
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Generative models for open domain question answering have proven to be competitive, without resorting to external knowledge.
Ranked #1 on
Question Answering
on ConditionalQA
Training with Quantization Noise for Extreme Model Compression
A standard solution is to train networks with Quantization Aware Training, where the weights are quantized during training and the gradients approximated with the Straight-Through Estimator.
Addressing Some Limitations of Transformers with Feedback Memory
Transformers have been successfully applied to sequential, auto-regressive tasks despite being feedforward networks.
End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
We study pseudo-labeling for the semi-supervised training of ResNet, Time-Depth Separable ConvNets, and Transformers for speech recognition, with either CTC or Seq2Seq loss functions.
Ranked #19 on
Speech Recognition
on LibriSpeech test-other
(using extra training data)
Unsupervised Cross-lingual Representation Learning at Scale
We also present a detailed empirical analysis of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale.
CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
Pre-training text representations have led to significant improvements in many areas of natural language processing.
Depth-Adaptive Transformer
State of the art sequence-to-sequence models for large scale tasks perform a fixed number of computations for each input sequence regardless of whether it is easy or hard to process.
Updating Pre-trained Word Vectors and Text Classifiers using Monolingual Alignment
In this paper, we focus on the problem of adapting word vector-based models to new textual data.
Lossless Data Compression with Transformer
It is closely related to the problem of online learning of language models.
Reducing Transformer Depth on Demand with Structured Dropout
Overparameterized transformer networks have obtained state of the art results in various natural language processing tasks, such as machine translation, language modeling, and question answering.
Ranked #5 on
Open-Domain Question Answering
on ELI5
Don't Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Induction
Human translators routinely have to translate rare inflections of words - due to the Zipfian distribution of words in a language.
Augmenting Self-attention with Persistent Memory
More precisely, we augment the self-attention layers with persistent memory vectors that play a similar role as the feed-forward layer.
Ranked #5 on
Language Modelling
on Text8
Training Hybrid Language Models by Marginalizing over Segmentations
In this paper, we study the problem of hybrid language modeling, that is using models which can predict both characters and larger units such as character ngrams or words.
Misspelling Oblivious Word Embeddings
In this paper we present a method to learn word embeddings that are resilient to misspellings.
Adaptive Attention Span in Transformers
We propose a novel self-attention mechanism that can learn its optimal attention span.
Ranked #4 on
Language Modelling
on Text8
Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling
Work on the problem of contextualized word representation—the development of reusable neural network components for sentence understanding—has recently seen a surge of progress centered on the unsupervised pretraining task of language modeling with methods like ELMo (Peters et al., 2018).
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling
Natural language understanding has recently seen a surge of progress with the use of sentence encoders like ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2019) which are pretrained on variants of language modeling.
Unsupervised Hyperalignment for Multilingual Word Embeddings
This paper extends this line of work to the problem of aligning multiple languages to a common space.
Unsupervised Alignment of Embeddings with Wasserstein Procrustes
A library for Multilingual Unsupervised or Supervised word Embeddings
Lightweight Adaptive Mixture of Neural and N-gram Language Models
It is often the case that the best performing language model is an ensemble of a neural language model with n-grams.
Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion
Continuous word representations learned separately on distinct languages can be aligned so that their words become comparable in a common space.
Colorless green recurrent networks dream hierarchically
Recurrent neural networks (RNNs) have achieved impressive results in a variety of linguistic processing tasks, suggesting that they can induce non-trivial properties of language.
Learning Word Vectors for 157 Languages
Distributed word representations, or word vectors, have recently been applied to many tasks in natural language processing, leading to state-of-the-art performance.
Ranked #12 on
Only Connect Walls Dataset Task 1 (Grouping)
on OCW
(using extra training data)
Advances in Pre-Training Distributed Word Representations
Many Natural Language Processing applications nowadays rely on pre-trained word representations estimated from large text corpora such as news collections, Wikipedia and Web Crawl.
Unbounded cache model for online language modeling with open vocabulary
Recently, continuous cache models were proposed as extensions to recurrent neural network language models, to adapt their predictions to local changes in the data distribution.
Fast Linear Model for Knowledge Graph Embeddings
This paper shows that a simple baseline based on a Bag-of-Words (BoW) representation learns surprisingly good knowledge graph embeddings.
Parseval Networks: Improving Robustness to Adversarial Examples
We introduce Parseval networks, a form of deep neural networks in which the Lipschitz constant of linear, convolutional and aggregation layers is constrained to be smaller than 1.
Improving Neural Language Models with a Continuous Cache
We propose an extension to neural network language models to adapt their prediction to the recent history.
Ranked #33 on
Language Modelling
on WikiText-2
FastText.zip: Compressing text classification models
We consider the problem of producing compact architectures for text classification, such that the full model fits in a limited amount of memory.
Variable Computation in Recurrent Neural Networks
Recurrent neural networks (RNNs) have been used extensively and with increasing success to model various types of sequential data.
Efficient softmax approximation for GPUs
We propose an approximate strategy to efficiently train neural network based language models over very large vocabularies.
Enriching Word Vectors with Subword Information
A vector representation is associated to each character $n$-gram; words being represented as the sum of these representations.
Ranked #3 on
Word Similarity
on WS353
Bag of Tricks for Efficient Text Classification
This paper explores a simple and efficient baseline for text classification.
Ranked #1 on
Sentiment Analysis
on Sogou News
Emotion Recognition in Conversation
General Classification
+2
Longitudinal Analysis of Discussion Topics in an Online Breast Cancer Community using Convolutional Neural Networks
Identifying topics of discussions in online health communities (OHC) is critical to various applications, but can be difficult because topics of OHC content are usually heterogeneous and domain-dependent.
Weakly-Supervised Alignment of Video With Text
Given vectorial features for both video and text, we propose to cast this task as a temporal assignment problem, with an implicit linear mapping between the two feature modalities.
Domain adaptation for sequence labeling using hidden Markov models
Most natural language processing systems based on machine learning are not robust to domain shift.
Trace Lasso: a trace norm regularization for correlated designs
This norm, called the trace Lasso, uses the trace norm of the selected covariates, which is a convex surrogate of their rank, as the criterion of model complexity.
