Search Results for author:
Found 29 papers, 18 papers with code
Transformer Quality in Linear Time
We revisit the design choices in Transformers, and propose methods to address their weaknesses in handling long sequences.
Ranked #1 on
Language Modelling
on Wiki-40B
Searching for Efficient Transformers for Language Modeling
For example, at a 500M parameter size, Primer improves the original T5 architecture on C4 auto-regressive language modeling, reducing the training cost by 4X.
Combined Scaling for Zero-shot Transfer Learning
Second, while increasing the dataset size and the model size has been the defacto method to improve the performance of deep learning models like BASIC, the effect of a large contrastive batch size on such contrastive-trained image-text models is not well-understood.
Primer: Searching for Efficient Transformers for Language Modeling
For example, at a 500M parameter size, Primer improves the original T5 architecture on C4 auto-regressive language modeling, reducing the training cost by 4X.
Ranked #1 on
Language Modelling
on C4
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
With recent progress in joint modeling of visual and textual representations, Vision-Language Pretraining (VLP) has achieved impressive performance on many multimodal downstream tasks.
Ranked #4 on
Visual Entailment
on SNLI-VE val
Combiner: Full Attention Transformer with Sparse Computation Cost
However, the key limitation of transformers is their quadratic memory and time complexity $\mathcal{O}(L^2)$ with respect to the sequence length in attention layers, which restricts application in extremely long sequences.
Ranked #2 on
Language Modelling
on Wiki-40B
CoAtNet: Marrying Convolution and Attention for All Data Sizes
Transformers have attracted increasing interests in computer vision, but they still fall behind state-of-the-art convolutional networks.
Ranked #1 on
Image Classification
on GasHisSDB
Pay Attention to MLPs
Transformers have become one of the most important architectural innovations in deep learning and have enabled many breakthroughs over the past few years.
Ranked #22 on
Natural Language Inference
on MultiNLI
Unsupervised Parallel Corpus Mining on Web Data
In contrast, there is a large-scale of parallel corpus created by humans on the Internet.
Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
With the success of language pretraining, it is highly desirable to develop more efficient architectures of good scalability that can exploit the abundant unlabeled data at a lower cost.
Ranked #6 on
Reading Comprehension
on RACE
Wiki-40B: Multilingual Language Model Dataset
We released the cleaned-up text of 40+ Wikipedia language editions, the corresponding trained monolingual language models, and several multilingual language models with different fixed vocabulary sizes.
Meta Pseudo Labels
We present Meta Pseudo Labels, a semi-supervised learning method that achieves a new state-of-the-art top-1 accuracy of 90. 2% on ImageNet, which is 1. 6% better than the existing state-of-the-art.
A Mutual Information Maximization Perspective of Language Representation Learning
We show state-of-the-art word representation learning methods maximize an objective function that is a lower bound on the mutual information between different parts of a word sequence (i. e., a sentence).
XLNet: Generalized Autoregressive Pretraining for Language Understanding
With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling.
Unsupervised Data Augmentation for Consistency Training
In this work, we present a new perspective on how to effectively noise unlabeled examples and argue that the quality of noising, specifically those produced by advanced data augmentation methods, plays a crucial role in semi-supervised learning.
Ranked #1 on
Sentiment Analysis
on Amazon Review Full
Re-examination of the Role of Latent Variables in Sequence Modeling
With latent variables, stochastic recurrent models have achieved state-of-the-art performance in modeling sound-wave sequence.
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling.
Ranked #3 on
Language Modelling
on One Billion Word
Characterizing and Avoiding Negative Transfer
When labeled data is scarce for a specific target task, transfer learning often offers an effective solution by utilizing data from a related source task.
Fast and Simple Mixture of Softmaxes with BPE and Hybrid-LightRNN for Language Generation
Mixture of Softmaxes (MoS) has been shown to be effective at addressing the expressiveness limitation of Softmax-based models.
Ranked #18 on
Machine Translation
on WMT2014 English-French
SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation
In this work, we examine methods for data augmentation for text-based tasks such as neural machine translation (NMT).
From Credit Assignment to Entropy Regularization: Two New Algorithms for Neural Sequence Prediction
In this work, we study the credit assignment problem in reward augmented maximum likelihood (RAML) learning, and establish a theoretical equivalence between the token-level counterpart of RAML and the entropy regularized reinforcement learning.
Large-scale Cloze Test Dataset Designed by Teachers
Cloze test is widely adopted in language exams to evaluate students' language proficiency.
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
We formulate language modeling as a matrix factorization problem, and show that the expressiveness of Softmax-based models (including the majority of neural language models) is limited by a Softmax bottleneck.
Ranked #10 on
Language Modelling
on Penn Treebank (Word Level)
Large-scale Cloze Test Dataset Created by Teachers
Cloze tests are widely adopted in language exams to evaluate students' language proficiency.
Controllable Invariance through Adversarial Feature Learning
Learning meaningful representations that maintain the content necessary for a particular task while filtering away detrimental variations is a problem of great interest in machine learning.
Good Semi-supervised Learning that Requires a Bad GAN
Semi-supervised learning methods based on generative adversarial networks (GANs) obtained strong empirical results, but it is not clear 1) how the discriminator benefits from joint training with a generator, and 2) why good semi-supervised classification performance and a good generator cannot be obtained at the same time.
An Interpretable Knowledge Transfer Model for Knowledge Base Completion
Knowledge bases are important resources for a variety of natural language processing tasks but suffer from incompleteness.
Calibrating Energy-based Generative Adversarial Networks
In this paper, we propose to equip Generative Adversarial Networks with the ability to produce direct energy estimates for samples. Specifically, we propose a flexible adversarial training framework, and prove this framework not only ensures the generator converges to the true data distribution, but also enables the discriminator to retain the density information at the global optimal.
Ranked #17 on
Conditional Image Generation
on CIFAR-10
(Inception score metric)
CFO: Conditional Focused Neural Question Answering with Large-scale Knowledge Bases
We propose CFO, a Conditional Focused neural-network-based approach to answering factoid questions with knowledge bases.
