Search Results for author:
Found 26 papers, 13 papers with code
Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
Sparse Mixture-of-Experts (MoE) is a neural architecture design that can be utilized to add learnable parameters to Large Language Models (LLMs) without increasing inference cost.
Scaling Instruction-Finetuned Language Models
We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation).
Ranked #1 on
Coreference Resolution
on WSC
Cross-Lingual Question Answering
Multi-task Language Understanding
+1
A Review of Sparse Expert Models in Deep Learning
Sparse expert models are a thirty-year old concept re-emerging as a popular architecture in deep learning.
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
There have been a lot of interest in the scaling properties of Transformer models.
Emergent Abilities of Large Language Models
Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
ST-MoE: Designing Stable and Transferable Sparse Expert Models
But advancing the state-of-the-art across a broad set of natural language tasks has been hindered by training instabilities and uncertain quality during fine-tuning.
Ranked #1 on
Common Sense Reasoning
on ARC (Easy)
Scale Efficiently: Insights from Pretraining and Finetuning Transformers
The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient.
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient.
On Bonus-Based Exploration Methods in the Arcade Learning Environment
Research on exploration in reinforcement learning, as applied to Atari 2600 game-playing, has emphasized tackling difficult exploration problems such as Montezuma's Revenge (Bellemare et al., 2016).
Revisiting ResNets: Improved Training and Scaling Strategies
Using improved training and scaling strategies, we design a family of ResNet architectures, ResNet-RS, which are 1. 7x - 2. 7x faster than EfficientNets on TPUs, while achieving similar accuracies on ImageNet.
Ranked #1 on
Document Image Classification
on AIP
Do Transformer Modifications Transfer Across Implementations and Applications?
The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread adoption.
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
We design models based off T5-Base and T5-Large to obtain up to 7x increases in pre-training speed with the same computational resources.
Revisiting Fundamentals of Experience Replay
Experience replay is central to off-policy algorithms in deep reinforcement learning (RL), but there remain significant gaps in our understanding.
On Catastrophic Interference in Atari 2600 Games
Our study provides a clear empirical link between catastrophic interference and sample efficiency in reinforcement learning.
On Bonus Based Exploration Methods In The Arcade Learning Environment
Research on exploration in reinforcement learning, as applied to Atari 2600 game-playing, has emphasized tackling difficult exploration problems such as Montezuma's Revenge (Bellemare et al., 2016).
Algorithmic Improvements for Deep Reinforcement Learning applied to Interactive Fiction
Empirically, we find that these techniques improve the performance of a baseline deep reinforcement learning agent applied to text-based games.
Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment
This paper provides an empirical evaluation of recently developed exploration algorithms within the Arcade Learning Environment (ALE).
Hyperbolic Discounting and Learning over Multiple Horizons
Reinforcement learning (RL) typically defines a discount factor as part of the Markov Decision Process.
Language GANs Falling Short
Generating high-quality text with sufficient diversity is essential for a wide range of Natural Language Generation (NLG) tasks.
Deep Graph Infomax
We present Deep Graph Infomax (DGI), a general approach for learning node representations within graph-structured data in an unsupervised manner.
Ranked #48 on
Node Classification
on Citeseer
Recall Traces: Backtracking Models for Efficient Reinforcement Learning
In many environments only a tiny subset of all states yield high reward.
Disentangling the independently controllable factors of variation by interacting with the world
It has been postulated that a good representation is one that disentangles the underlying explanatory factors of variation.
MaskGAN: Better Text Generation via Filling in the______
Additionally, these models are typically trained via maxi- mum likelihood and teacher forcing.
MaskGAN: Better Text Generation via Filling in the _______
Neural autoregressive and seq2seq models that generate text by sampling words sequentially, with each word conditioned on the previous model, are state-of-the-art for several machine translation and summarization benchmarks.
Ranked #4 on
Multivariate Time Series Imputation
on PEMS-SF
Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Step
Unlike other generative models, the data distribution is learned via a game between a generator (the generative model) and a discriminator (a teacher providing training signal) that each minimize their own cost.
