Search Results for author:
Found 112 papers, 43 papers with code
PaLI-X: On Scaling up a Multilingual Vision and Language Model
We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture.
PaLM 2 Technical Report
Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM.
Symbol tuning improves in-context learning in language models
We present symbol tuning - finetuning language models on in-context input-label pairs where natural language labels (e. g., "positive/negative sentiment") are replaced with arbitrary symbols (e. g., "foo/bar").
Recommender Systems with Generative Retrieval
Once we have the Semantic IDs for all the items, a Transformer based sequence-to-sequence model is trained to predict the Semantic ID of the next item.
UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining
As part of our contribution, we release: (i) an improved and refreshed mC4 multilingual corpus consisting of 29 trillion characters across 107 languages, and (ii) a suite of pretrained umT5 model checkpoints trained with UniMax sampling.
CoLT5: Faster Long-Range Transformers with Conditional Computation
Many natural language processing tasks benefit from long inputs, but processing long documents with Transformers is expensive -- not only due to quadratic attention complexity but also from applying feedforward and projection layers to every token.
Ranked #1 on
Long-range modeling
on SCROLLS
Larger language models do in-context learning differently
We next study semantically-unrelated label ICL (SUL-ICL), in which labels are semantically unrelated to their inputs (e. g., foo/bar instead of negative/positive), thereby forcing language models to learn the input-label mappings shown in in-context exemplars in order to perform the task.
Scaling Vision Transformers to 22 Billion Parameters
The scaling of Transformers has driven breakthrough capabilities for language models.
Ranked #1 on
Linear-Probe Classification
on ImageNet
(using extra training data)
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
We study the design decisions of publicly available instruction tuning methods, and break down the development of Flan 2022 (Chung et al., 2022).
DSI++: Updating Transformer Memory with New Documents
In this work, we introduce DSI++, a continual learning challenge for DSI to incrementally index new documents while being able to answer queries related to both previously and newly indexed documents.
Dense Feature Memory Augmented Transformers for COVID-19 Vaccination Search Classification
With the devastating outbreak of COVID-19, vaccines are one of the crucial lines of defense against mass infection in this global pandemic.
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
In this work, we propose sparse upcycling -- a simple way to reuse sunk training costs by initializing a sparsely activated Mixture-of-Experts model from a dense checkpoint.
Inverse scaling can become U-shaped
The Inverse Scaling Prize (McKenzie et al. 2022) identified eleven such inverse scaling tasks, evaluated on models of up to 280B parameters and up to 500 zettaFLOPs of training compute.
Scaling Instruction-Finetuned Language Models
We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation).
Ranked #1 on
Coreference Resolution
on WSC
Cross-Lingual Question Answering
Multi-task Language Understanding
+1
Transcending Scaling Laws with 0.1% Extra Compute
This paper proposes UL2R, a method that substantially improves existing language models and their scaling curves with a relatively tiny amount of extra compute.
Ranked #2 on
Cross-Lingual Question Answering
on TyDiQA-GoldP
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
BIG-Bench (Srivastava et al., 2022) is a diverse evaluation suite that focuses on tasks believed to be beyond the capabilities of current language models.
Language Models are Multilingual Chain-of-Thought Reasoners
Finally, we show that the multilingual reasoning abilities of language models extend to other tasks such as commonsense reasoning and word-in-context semantic judgment.
Recitation-Augmented Language Models
We propose a new paradigm to help Large Language Models (LLMs) generate more accurate factual knowledge without retrieving from an external corpus, called RECITation-augmented gEneration (RECITE).
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
There have been a lot of interest in the scaling properties of Transformer models.
Confident Adaptive Language Modeling
Recent advances in Transformer-based large language models (LLMs) have led to significant performance improvements across many tasks.
Emergent Abilities of Large Language Models
Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks.
UL2: Unifying Language Learning Paradigms
Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization.
Ranked #7 on
Long-range modeling
on SCROLLS
ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference
This results in significant inference time speedups since the decoder-only architecture only needs to learn to interpret static encoder embeddings during inference.
PaLM: Scaling Language Modeling with Pathways
To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM.
Ranked #1 on
Common Sense Reasoning
on BIG-bench (Known Unknowns)
HyperPrompt: Prompt-based Task-Conditioning of Transformers
Prompt-Tuning is a new paradigm for finetuning pre-trained language models in a parameter-efficient way.
A New Generation of Perspective API: Efficient Multilingual Character-level Transformers
As such, it is crucial to develop models that are effective across a diverse range of languages, usages, and styles.
Transformer Memory as a Differentiable Search Index
In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of the model.
Self-Instantiated Recurrent Units with Dynamic Soft Recursion
On the other hand, the extent of the Self-IRU recursion is controlled by gates whose values are between 0 and 1 and may vary across the temporal dimension of sequences, enabling dynamic soft recursion depth at each time step.
PolyViT: Co-training Vision Transformers on Images, Videos and Audio
Can we train a single transformer model capable of processing multiple modalities and datasets, whilst sharing almost all of its learnable parameters?
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
Despite the recent success of multi-task learning and transfer learning for natural language processing (NLP), few works have systematically studied the effect of scaling up the number of tasks during pre-training.
SCENIC: A JAX Library for Computer Vision Research and Beyond
Scenic is an open-source JAX library with a focus on Transformer-based models for computer vision research and beyond.
Improving Compositional Generalization with Self-Training for Data-to-Text Generation
Data-to-text generation focuses on generating fluent natural language responses from structured meaning representations (MRs).
Sharpness-Aware Minimization Improves Language Model Generalization
The allure of superhuman-level capabilities has led to considerable interest in language models like GPT-3 and T5, wherein the research has, by and large, revolved around new model architectures, training tasks, and loss objectives, along with substantial engineering efforts to scale up model capacity and dataset size.
Improving Neural Ranking via Lossless Knowledge Distillation
We explore a novel perspective of knowledge distillation (KD) for learning to rank (LTR), and introduce Self-Distilled neural Rankers (SDR), where student rankers are parameterized identically to their teachers.
Scale Efficiently: Insights from Pretraining and Finetuning Transformers
The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient.
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient.
On Orthogonality Constraints for Transformers
Orthogonality constraints encourage matrices to be orthogonal for numerical stability.
Are Pretrained Convolutions Better than Pretrained Transformers?
In the context of language models, are convolutional models competitive to Transformers when pre-trained?
The Benchmark Lottery
The world of empirical machine learning (ML) strongly relies on benchmarks in order to determine the relative effectiveness of different algorithms and methods.
SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
Self-supervised contrastive representation learning has proved incredibly successful in the vision and natural language domains, enabling state-of-the-art performance with orders of magnitude less labeled data.
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
In this paper, we propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model.
Ranked #3 on
Paraphrase Identification
on Quora Question Pairs
Knowledge Router: Learning Disentangled Representations for Knowledge Graphs
To this end, this paper proposes to learn disentangled representations of KG entities - a new method that disentangles the inner latent properties of KG entities.
How Reliable are Model Diagnostics?
In the pursuit of a deeper understanding of a model's behaviour, there is recent impetus for developing suites of probes aimed at diagnosing models beyond simple metrics like accuracy or BLEU.
Are Pre-trained Convolutions Better than Pre-trained Transformers?
In the context of language models, are convolutional models competitive to Transformers when pre-trained?
Rethinking Search: Making Domain Experts out of Dilettantes
When experiencing an information need, users want to engage with a domain expert, but often turn to an information retrieval system, such as a search engine, instead.
OmniNet: Omnidirectional Representations from Transformers
In OmniNet, instead of maintaining a strictly horizontal receptive field, each token is allowed to attend to all tokens in the entire network.
Ranked #1 on
Machine Translation
on WMT2017 Russian-English
Do Transformer Modifications Transfer Across Implementations and Applications?
The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread adoption.
Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with $1/n$ Parameters
Recent works have demonstrated reasonable success of representation learning in hypercomplex space.
Switch Spaces: Learning Product Spaces with Sparse Gating
In order for learned representations to be effective and efficient, it is ideal that the geometric inductive bias aligns well with the underlying structure of the data.
Label Smoothed Embedding Hypothesis for Out-of-Distribution Detection
Detecting out-of-distribution (OOD) examples is critical in many applications.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
HyperGrid Transformers: Towards A Single Model for Multiple Tasks
Specifically, we propose a decomposable hypernetwork that learns grid-wise projections that help to specialize regions in weight matrices for different tasks.
Long Range Arena : A Benchmark for Efficient Transformers
Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity.
Parameterization of Hypercomplex Multiplications
Recent works have demonstrated reasonable success of representation learning in hypercomplex space.
Neural Rankers are hitherto Outperformed by Gradient Boosted Decision Trees
We first validate this concern by showing that most recent neural LTR models are, by a large margin, inferior to the best publicly available Gradient Boosted Decision Trees (GBDT) in terms of their reported ranking accuracy on benchmark datasets.
Synthesizer: Rethinking Self-Attention for Transformer Models
The dot product self-attention is known to be central and indispensable to state-of-the-art Transformer models.
Recurrently Controlling a Recurrent Network with Recurrent Networks Controlled by More Recurrent Networks
This paper explores an intriguing idea of recursively parameterizing recurrent nets.
StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling
There are two major classes of natural language grammar -- the dependency grammar that models one-to-one correspondences between words and the constituency grammar that models the assembly of one or several corresponded words.
Long Range Arena: A Benchmark for Efficient Transformers
In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality to vanilla Transformer models.
Ranked #15 on
Long-range modeling
on LRA
Surprise: Result List Truncation via Extreme Value Theory
Work in information retrieval has largely been centered around ranking and relevance: given a query, return some number of results ordered by relevance to the user.
Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencoder
This paper demonstrates a fatal vulnerability in natural language inference (NLI) and text classification systems.
Efficient Transformers: A Survey
Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision and reinforcement learning.
Generative Models are Unsupervised Predictors of Page Quality: A Colossal-Scale Study
Large generative language models such as GPT-2 are well-known for their ability to generate text as well as their utility in supervised downstream tasks via fine-tuning.
HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections
The proposed approach is based on a decomposable hypernetwork that learns grid-wise projections that help to specialize regions in weight matrices for different tasks.
Would you Rather? A New Benchmark for Learning Machine Alignment with Cultural Values and Social Preferences
Understanding human preferences, along with cultural and social nuances, lives at the heart of natural language understanding.
Synthesizer: Rethinking Self-Attention in Transformer Models
The dot product self-attention is known to be central and indispensable to state-of-the-art Transformer models.
Ranked #1 on
Dialogue Generation
on Persona-Chat
(BLEU-1 metric)
Choppy: Cut Transformer For Ranked List Truncation
Work in information retrieval has traditionally focused on ranking and relevance: given a query, return some number of results ordered by relevance to the user.
Reverse Engineering Configurations of Neural Text Generation Models
This paper seeks to develop a deeper understanding of the fundamental properties of neural text generations models.
Sparse Sinkhorn Attention
We propose Sparse Sinkhorn Attention, a new efficient and sparse method for learning to attend.
Multi-level Head-wise Match and Aggregation in Transformer for Textual Sequence Matching
Transformer has been successfully applied to many natural language processing tasks.
Metagross: Meta Gated Recursive Controller Units for Sequence Modeling
This paper proposes Metagross (Meta Gated Recursive Controller), a new neural sequence modeling unit.
Jacobian Adversarially Regularized Networks for Robustness
Adversarial examples are crafted with imperceptible perturbations with the intent to fool neural networks.
What it Thinks is Important is Important: Robustness Transfers through Input Gradients
Learned weights of models robust to such perturbations are previously found to be transferable across different tasks but this applies only if the model architecture for the source and target tasks is the same.
Compositional De-Attention Networks
Attentional models are distinctly characterized by their ability to learn relative importance, i. e., assigning a different weight to input values.
Bridging the Gap between Relevance Matching and Semantic Matching for Short Text Similarity Modeling
A core problem of information retrieval (IR) is relevance matching, which is to rank documents by relevance to a user{'}s query.
R2D2: Reuse & Reduce via Dynamic Weight Diffusion for Training Efficient NLP Models
We propose R2D2 layers, a new neural block for training efficient NLP models.
Interactive Machine Comprehension with Information Seeking Agents
Existing machine reading comprehension (MRC) models do not scale effectively to real-world applications like web-level information retrieval and question answering (QA).
Holographic Factorization Machines for Recommendation
Factorization Machines (FMs) are a class of popular algorithms that have been widely adopted for collaborative filtering and recommendation tasks.
Confusionset-guided Pointer Networks for Chinese Spelling Check
This paper proposes Confusionset-guided Pointer Networks for Chinese Spell Check (CSC) task.
Robust Representation Learning of Biomedical Names
Moreover, our proposed method is also able to compute meaningful representations for unseen names, resulting in high practical utility in real-world applications.
Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks
Many state-of-the-art neural models for NLP are heavily parameterized and thus memory inefficient.
Quaternion Collaborative Filtering for Recommendation
All in all, we conduct extensive experiments on six real-world datasets, demonstrating the effectiveness of Quaternion algebra in recommender systems.
Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narratives
This paper tackles the problem of reading comprehension over long narratives where documents easily span over thousands of tokens.
DeepRec: An Open-source Toolkit for Deep Learning based Recommendation
In this toolkit, we have implemented a number of deep learning based recommendation algorithms using Python and the widely used deep learning package - Tensorflow.
Quaternion Knowledge Graph Embeddings
In this work, we move beyond the traditional complex-valued representations, introducing more expressive hypercomplex representations to model entities and relations for knowledge graph embeddings.
Ranked #4 on
Link Prediction
on FB15k
Recurrently Controlled Recurrent Networks
Recurrent neural networks (RNNs) such as long short-term memory and gated recurrent units are pivotal building blocks across a broad spectrum of sequence modeling problems.
Holistic Multi-modal Memory Network for Movie Question Answering
In this paper, we present the Holistic Multi-modal Memory Network (HMMN) framework which fully considers the interactions between different input sources (multi-modal context, question) in each hop.
Densely Connected Attention Propagation for Reading Comprehension
Secondly, the dense connectors in our network are learned via attention instead of standard residual skip-connectors.
Ranked #2 on
Question Answering
on NewsQA
Co-Stack Residual Affinity Networks with Multi-level Attention Refinement for Matching Text Sequences
This task enables many potential applications such as question answering and paraphrase identification.
Multi-Granular Sequence Encoding via Dilated Compositional Units for Reading Comprehension
Sequence encoders are crucial components in many neural architectures for learning to read and comprehend.
Ranked #7 on
Question Answering
on NarrativeQA
Attentive Gated Lexicon Reader with Contrastive Contextual Co-Attention for Sentiment Classification
This paper proposes a new neural architecture that exploits readily available sentiment lexicon resources.
HyperML: A Boosting Metric Learning Approach in Hyperbolic Space for Recommender Systems
This paper investigates the notion of learning user and item representations in non-Euclidean space.
Ranked #1 on
Recommendation Systems
on MovieLens 20M
(HR@10 metric)
Next Item Recommendation with Self-Attention
In this paper, we propose a novel sequence-aware recommendation model.
Self-Attentive Neural Collaborative Filtering
This paper has been withdrawn as we discovered a bug in our tensorflow implementation that involved accidental mixing of vectors across batches.
Multi-Cast Attention Networks for Retrieval-based Question Answering and Response Prediction
Attention is typically used to select informative sub-phrases that are used for prediction.
CoupleNet: Paying Attention to Couples with Coupled Attention for Relationship Recommendation
Our approach, the CoupleNet is an end-to-end deep learning based estimator that analyzes the social profiles of two users and subsequently performs a similarity match between the users.
Reasoning with Sarcasm by Reading In-between
Sarcasm is a sophisticated speech act which commonly manifests on social communities such as Twitter and Reddit.
Interact and Decide: Medley of Sub-Attention Networks for Effective Group Recommendation
Our proposed approach hinges upon the key intuition that the decision making process (in groups) is generally dynamic, i. e., a user's decision is highly dependent on the other group members.
Multi-range Reasoning for Machine Comprehension
Similarly, we achieve competitive performance relative to AMANDA on the SearchQA benchmark and BiDAF on the NarrativeQA benchmark without using any LSTM/GRU layers.
Ranked #5 on
Question Answering
on RACE
Metric Factorization: Recommendation beyond Matrix Factorization
In the past decade, matrix factorization has been extensively researched and has become one of the most popular techniques for personalized recommendations.
Pair-Linking for Collective Entity Disambiguation: Two Could Be Better Than All
For the first time, we show that the semantic relationships between the mentioned entities are in fact less dense than expected.
Multi-Pointer Co-Attention Networks for Recommendation
Our model operates on a multi-hierarchical paradigm and is based on the intuition that not all reviews are created equal, i. e., only a select few are important.
Compare, Compress and Propagate: Enhancing Neural Architectures with Alignment Factorization for Natural Language Inference
Firstly, we introduce a new architecture where alignment pairs are compared, compressed and then propagated to upper layers for enhanced representation learning.
Ranked #7 on
Natural Language Inference
on SciTail
Learning to Attend via Word-Aspect Associative Fusion for Aspect-based Sentiment Analysis
Our novel model, \textit{Aspect Fusion LSTM} (AF-LSTM) learns to attend based on associative relationships between sentence words and aspect which allows our model to adaptively focus on the correct words given an aspect term.
Cross Temporal Recurrent Networks for Ranking Question Answer Pairs
This paper explores the idea of learning temporal gates for sequence pairs (question and answer), jointly influencing the learned representations in a pairwise manner.
SkipFlow: Incorporating Neural Coherence Features for End-to-End Automatic Text Scoring
Our new method proposes a new \textsc{SkipFlow} mechanism that models relationships between snapshots of the hidden representations of a long short-term memory (LSTM) network as it reads.
Ranked #4 on
Automated Essay Scoring
on ASAP
Multi-task Neural Network for Non-discrete Attribute Prediction in Knowledge Graphs
Unfortunately, many state-of-the-art relational learning models ignore this information due to the challenging nature of dealing with non-discrete data types in the inherently binary-natured knowledge graphs.
Hyperbolic Representation Learning for Fast and Efficient Neural Question Answering
The dominant neural architectures in question answer retrieval are based on recurrent or convolutional encoders configured with complex word matching layers.
Ranked #1 on
Question Answering
on SemEvalCQA
Deep Learning based Recommender System: A Survey and New Perspectives
This article aims to provide a comprehensive review of recent research efforts on deep learning based recommender systems.
Latent Relational Metric Learning via Memory-based Attention for Collaborative Ranking
Our model, LRML (\textit{Latent Relational Metric Learning}) is a novel metric learning approach for recommendation.
Ranked #1 on
Recommendation Systems
on Netflix
(nDCG@10 metric)
Cross Device Matching for Online Advertising with Neural Feature Ensembles : First Place Solution at CIKM Cup 2016
We describe the 1st place winning approach for the CIKM Cup 2016 Challenge.
