Search Results for author:
Found 12 papers, 8 papers with code
Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
This paper explores the effectiveness of model-generated signals in improving zero-shot generalization of text-to-text Transformers such as T5.
Understanding the Effectiveness of Very Large Language Models on Dialog Evaluation
The paper shows that the choice of datasets used for training a model contributes to how well it performs on a task as well as on how the prompt should be structured.
Foundation Transformers
A big convergence of model architectures across language, vision, speech, and multimodal is emerging.
On the Representation Collapse of Sparse Mixture of Experts
We also present a comprehensive analysis on the representation and routing behaviors of our models.
METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model.
Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators
We present a new framework AMOS that pretrains text encoders with an Adversarial learning curriculum via a Mixture Of Signals from multiple auxiliary generators.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
In this paper, we introduce ELECTRA-style tasks to cross-lingual language model pre-training.
Ranked #1 on
Zero-Shot Cross-Lingual Transfer
on XTREME
Language Scaling for Universal Suggested Replies Model
Faced with increased compute requirements and low resources for language expansion, we build a single universal model for improving the quality and reducing run-time costs of our production system.
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
The first token-level task, Corrective Language Modeling, is to detect and correct tokens replaced by the auxiliary model, in order to better capture token-level semantics.
Embedding Logical Queries on Knowledge Graphs
Learning low-dimensional embeddings of knowledge graphs is a powerful approach used to predict unobserved or missing edges between entities.
Ranked #6 on
Complex Query Answering
on FB15k-237
Inferring Generative Model Structure with Static Analysis
Obtaining enough labeled data to robustly train complex discriminative models is a major bottleneck in the machine learning pipeline.
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
The size of the dataset and the fact that the questions are derived from real user search queries distinguishes MS MARCO from other well-known publicly available datasets for machine reading comprehension and question-answering.
