Search Results for author:
Found 51 papers, 15 papers with code
Multi-Task Learning for Email Search Ranking with Auxiliary Query Clustering
In this paper, we study how to obtain query type in an unsupervised fashion and how to incorporate this information into query-dependent ranking models.
Domain Adaptation for Enterprise Email Search
To address this data challenge, in this paper we propose a domain adaptation approach that fine-tunes the global model to each individual enterprise.
Separate and Attend in Personal Email Search
In personal email search, user queries often impose different requirements on different aspects of the retrieved emails.
Sparse Sinkhorn Attention
We propose Sparse Sinkhorn Attention, a new efficient and sparse method for learning to attend.
Reverse Engineering Configurations of Neural Text Generation Models
This paper seeks to develop a deeper understanding of the fundamental properties of neural text generations models.
Choppy: Cut Transformer For Ranked List Truncation
Work in information retrieval has traditionally focused on ranking and relevance: given a query, return some number of results ordered by relevance to the user.
Synthesizer: Rethinking Self-Attention in Transformer Models
The dot product self-attention is known to be central and indispensable to state-of-the-art Transformer models.
Ranked #1 on
Dialogue Generation
on Persona-Chat
(BLEU-1 metric, using extra
training data)
HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections
The proposed approach is based on a decomposable hypernetwork that learns grid-wise projections that help to specialize regions in weight matrices for different tasks.
Generative Models are Unsupervised Predictors of Page Quality: A Colossal-Scale Study
Large generative language models such as GPT-2 are well-known for their ability to generate text as well as their utility in supervised downstream tasks via fine-tuning.
Efficient Transformers: A Survey
Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision and reinforcement learning.
Surprise: Result List Truncation via Extreme Value Theory
Work in information retrieval has largely been centered around ranking and relevance: given a query, return some number of results ordered by relevance to the user.
Long Range Arena: A Benchmark for Efficient Transformers
In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality to vanilla Transformer models.
Ranked #18 on
Long-range modeling
on LRA
(Pathfinder metric)
StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling
There are two major classes of natural language grammar -- the dependency grammar that models one-to-one correspondences between words and the constituency grammar that models the assembly of one or several corresponded words.
Synthesizer: Rethinking Self-Attention for Transformer Models
The dot product self-attention is known to be central and indispensable to state-of-the-art Transformer models.
Long Range Arena : A Benchmark for Efficient Transformers
Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity.
HyperGrid Transformers: Towards A Single Model for Multiple Tasks
Specifically, we propose a decomposable hypernetwork that learns grid-wise projections that help to specialize regions in weight matrices for different tasks.
Label Smoothed Embedding Hypothesis for Out-of-Distribution Detection
Detecting out-of-distribution (OOD) examples is critical in many applications.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
OmniNet: Omnidirectional Representations from Transformers
In OmniNet, instead of maintaining a strictly horizontal receptive field, each token is allowed to attend to all tokens in the entire network.
Ranked #1 on
Machine Translation
on WMT2017 Russian-English
Rethinking Search: Making Domain Experts out of Dilettantes
When experiencing an information need, users want to engage with a domain expert, but often turn to an information retrieval system, such as a search engine, instead.
Are Pre-trained Convolutions Better than Pre-trained Transformers?
In the context of language models, are convolutional models competitive to Transformers when pre-trained?
How Reliable are Model Diagnostics?
In the pursuit of a deeper understanding of a model's behaviour, there is recent impetus for developing suites of probes aimed at diagnosing models beyond simple metrics like accuracy or BLEU.
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
In this paper, we propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model.
Ranked #3 on
Paraphrase Identification
on Quora Question Pairs
SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
Self-supervised contrastive representation learning has proved incredibly successful in the vision and natural language domains, enabling state-of-the-art performance with orders of magnitude less labeled data.
The Benchmark Lottery
The world of empirical machine learning (ML) strongly relies on benchmarks in order to determine the relative effectiveness of different algorithms and methods.
Are Pretrained Convolutions Better than Pretrained Transformers?
In the context of language models, are convolutional models competitive to Transformers when pre-trained?
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient.
Scale Efficiently: Insights from Pretraining and Finetuning Transformers
The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient.
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
Despite the recent success of multi-task learning and transfer learning for natural language processing (NLP), few works have systematically studied the effect of scaling up the number of tasks during pre-training.
Atomized Search Length: Beyond User Models
We argue that current IR metrics, modeled on optimizing user experience, measure too narrow a portion of the IR space.
Transformer Memory as a Differentiable Search Index
In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of the model.
A New Generation of Perspective API: Efficient Multilingual Character-level Transformers
As such, it is crucial to develop models that are effective across a diverse range of languages, usages, and styles.
HyperPrompt: Prompt-based Task-Conditioning of Transformers
Prompt-Tuning is a new paradigm for finetuning pre-trained language models in a parameter-efficient way.
Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters
Natural Language Inference (NLI) has been extensively studied by the NLP community as a framework for estimating the semantic relation between sentence pairs.
Retrieval-Enhanced Machine Learning
Although information access systems have long supported people in accomplishing a wide range of tasks, we propose broadening the scope of users of information access systems to include task-driven machines, such as machine learning models.
UL2: Unifying Language Learning Paradigms
Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization.
Ranked #1 on
Long-range modeling
on SCROLLS
(CNLI metric)
Emergent Abilities of Large Language Models
Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks.
Confident Adaptive Language Modeling
Recent advances in Transformer-based large language models (LLMs) have led to significant performance improvements across many tasks.
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
There have been a lot of interest in the scaling properties of Transformer models.
Transcending Scaling Laws with 0.1% Extra Compute
This paper proposes UL2R, a method that substantially improves existing language models and their scaling curves with a relatively tiny amount of extra compute.
Ranked #2 on
Cross-Lingual Question Answering
on TyDiQA-GoldP
Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models
We take human annotations as a gold standard and show that a correlated automatic metric is suitable for development.
Dense Feature Memory Augmented Transformers for COVID-19 Vaccination Search Classification
With the devastating outbreak of COVID-19, vaccines are one of the crucial lines of defense against mass infection in this global pandemic.
DSI++: Updating Transformer Memory with New Documents
In this work, we introduce DSI++, a continual learning challenge for DSI to incrementally index new documents while being able to answer queries related to both previously and newly indexed documents.
How Does Generative Retrieval Scale to Millions of Passages?
Popularized by the Differentiable Search Index, the emerging paradigm of generative retrieval re-frames the classic information retrieval problem into a sequence-to-sequence modeling task, forgoing external indices and encoding an entire document corpus within a single Transformer.
LAIT: Efficient Multi-Segment Encoding in Transformers with Layer-Adjustable Interaction
Transformer encoders contextualize token representations by attending to all other tokens at each layer, leading to quadratic increase in compute effort with the input length.
Gen-IR @ SIGIR 2023: The First Workshop on Generative Information Retrieval
Generative information retrieval (IR) has experienced substantial growth across multiple research communities (e. g., information retrieval, computer vision, natural language processing, and machine learning), and has been highly visible in the popular press.
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting
Ranking documents using Large Language Models (LLMs) by directly feeding the query and candidate documents into the prompt is an interesting and practical problem.
PaRaDe: Passage Ranking using Demonstrations with Large Language Models
Recent studies show that large language models (LLMs) can be instructed to effectively perform zero-shot passage re-ranking, in which the results of a first stage retrieval method, such as BM25, are rated and reordered to improve relevance.
SEMQA: Semi-Extractive Multi-Source Question Answering
Experimenting with several LLMs in various settings, we find this task to be surprisingly challenging, demonstrating the importance of QuoteSum for developing and studying such consolidation capabilities.
Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data
Reinforcement Learning from Human Feedback (RLHF) is a popular method for aligning Language Models (LM) with human values and preferences.
Impact of Preference Noise on the Alignment Performance of Generative Language Models
A key requirement in developing Generative Language Models (GLMs) is to have their values aligned with human values.
