Search Results for author:
Found 31 papers, 19 papers with code
Overview of the SustaiNLP 2020 Shared Task
We describe the SustaiNLP 2020 shared task: efficient inference on the SuperGLUE benchmark (Wang et al., 2019).
StarCoder 2 and The Stack v2: The Next Generation
Our large model, StarCoder2- 15B, significantly outperforms other models of comparable size.
Ranked #24 on
Code Generation
on MBPP
GAIA: a benchmark for General AI Assistants
GAIA's philosophy departs from the current trend in AI benchmarks suggesting to target tasks that are ever more difficult for humans.
FinGPT: Large Generative Models for a Small Language
We pursue two approaches to pretrain models: 1) we train seven monolingual models from scratch (186M to 13B parameters) dubbed FinGPT, 2) we continue the pretraining of the multilingual BLOOM model on a mix of its original training data and Finnish, resulting in a 176 billion parameter model we call BLUUMI.
Zephyr: Direct Distillation of LM Alignment
Starting from a dataset of outputs ranked by a teacher model, we apply distilled direct preference optimization (dDPO) to learn a chat model with significantly improved intent alignment.
Scaling Data-Constrained Language Models
We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data.
StarCoder: may the source be with you!
The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase: 15. 5B parameter models with 8K context length, infilling capabilities and fast large-batch inference enabled by multi-query attention.
Ranked #42 on
Code Generation
on MBPP
Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning
Using an interactive textual environment designed to study higher-level forms of functional grounding, and a set of spatial and navigation tasks, we study several scientific questions: 1) Can LLMs boost sample efficiency for online learning of various RL tasks?
The Stack: 3 TB of permissively licensed source code
Large Language Models (LLMs) play an ever-increasing role in the field of Artificial Intelligence (AI)--not only for natural language processing but also for code understanding and generation.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements
We introduce Evaluate and Evaluation on the Hub --a set of tools to facilitate the evaluation of models and datasets in ML.
Training Transformers Together
The infrastructure necessary for training state-of-the-art models is becoming overly expensive, which makes training such models affordable only to large corporations and institutions.
Multitask Prompted Training Enables Zero-Shot Task Generalization
Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020).
Datasets: A Community Library for Natural Language Processing
The scale, variety, and quantity of publicly-available NLP datasets has grown rapidly as researchers propose new tasks, larger models, and novel benchmarks.
VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning
Unlike language, where the text tokens are more independent, neighboring video tokens typically have strong correlations (e. g., consecutive video frames usually look very similar), and hence uniformly masking individual tokens will make the task too trivial to learn useful representations.
Ranked #10 on
Action Recognition
on Diving-48
Distributed Deep Learning in Open Collaborations
Modern deep learning applications require increasingly more compute to train state-of-the-art models.
Evolution of Helimagnetic Correlations when approaching the Quantum Critical Point of Mn$_{1-x}$Fe$_x$Si
We argue that this effect explains both the expansion of the precursor phase with increasing $x$ and the abrupt disappearance of long range helimagnetic periodicity at $x^*$.
Strongly Correlated Electrons Mesoscale and Nanoscale Physics
Learning from others' mistakes: Avoiding dataset biases without modeling them
State-of-the-art natural language processing (NLP) models often learn to model dataset biases and surface form correlations instead of features that target the intended underlying task.
The Amazing World of Neural Language Generation
Neural Language Generation (NLG) {--} using neural network models to generate coherent text {--} is among the most promising methods for automated text creation.
Transformers: State-of-the-Art Natural Language Processing
Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks.
Movement Pruning: Adaptive Sparsity by Fine-Tuning
Magnitude pruning is a widely used strategy for reducing model size in pure supervised learning; however, it is less effective in the transfer learning regime that has become standard for state-of-the-art natural language processing applications.
TLDR: Token Loss Dynamic Reweighting for Reducing Repetitive Utterance Generation
We hypothesize that the deeper reason is that in the training corpora, there are hard tokens that are more difficult for a generative model to learn than others and, once learning has finished, hard tokens are still under-learned, so that repetitive generations are more likely to happen.
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks.
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging.
Ranked #2 on
Only Connect Walls Dataset Task 1 (Grouping)
on OCW
(Wasserstein Distance (WD) metric, using extra
training data)
Large-Scale Transfer Learning for Natural Language Generation
Large-scale pretrained language models define state of the art in natural language processing, achieving outstanding performance on a variety of tasks.
Transfer Learning in Natural Language Processing
The classic supervised machine learning paradigm is based on learning in isolation, a single predictive model for a task using a single dataset.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
We introduce a new approach to generative data-driven dialogue systems (e. g. chatbots) called TransferTransfo which is a combination of a Transfer learning based training scheme and a high-capacity Transformer model.
Ranked #3 on
Dialogue Generation
on Persona-Chat
(using extra training data)
A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks
The model is trained in a hierarchical fashion to introduce an inductive bias by supervising a set of low level tasks at the bottom layers of the model and more complex tasks at the top layers of the model.
Ranked #10 on
Relation Extraction
on ACE 2005
(using extra training data)
Continuous Learning in a Hierarchical Multiscale Neural Network
We reformulate the problem of encoding a multi-scale representation of a sequence in a language model by casting it in a continuous learning framework.
Meta-Learning a Dynamical Language Model
We consider the task of word-level language modeling and study the possibility of combining hidden-states-based short-term representations with medium-term representations encoded in dynamical weights of a language model.
Studying Invariances of Trained Convolutional Neural Networks
Convolutional Neural Networks (CNNs) define an exceptionally powerful class of models for image classification, but the theoretical background and the understanding of how invariances to certain transformations are learned is limited.
