Search Results for author:
Found 26 papers, 15 papers with code
Language Is Not All You Need: Aligning Perception with Language Models
A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence.
Ranked #2 on
Image Captioning
on Flickr30k Captions test
(CIDEr metric)
TorchScale: Transformers at Scale
Large Transformers have achieved state-of-the-art performance across many tasks.
Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
In this paper, we elaborate upon recipes for building multilingual representation models that are not only competitive with existing state-of-the-art models but are also more parameter efficient, thereby promoting better adoption in resource-constrained scenarios and practical applications.
Foundation Transformers
A big convergence of model architectures across language, vision, speech, and multimodal is emerging.
On the Representation Collapse of Sparse Mixture of Experts
We also present a comprehensive analysis on the representation and routing behaviors of our models.
METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model.
Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators
We present a new framework AMOS that pretrains text encoders with an Adversarial learning curriculum via a Mixture Of Signals from multiple auxiliary generators.
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
Next, we detail the training process, the design of our training corpus, and our data curation techniques, which we believe is a key ingredient to the success of the model.
Ranked #2 on
Language Modelling
on LAMBADA
Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task
This report describes Microsoft's machine translation systems for the WMT21 shared task on large-scale multilingual machine translation.
Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
We find that many languages are under-represented in recent cross-lingual language models due to the limited vocabulary capacity.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
In this paper, we introduce ELECTRA-style tasks to cross-lingual language model pre-training.
Ranked #1 on
Zero-Shot Cross-Lingual Transfer
on XTREME
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
While pretrained encoders have achieved success in various natural language understanding (NLU) tasks, there is a gap between these pretrained encoders and natural language generation (NLG).
Consistency Regularization for Cross-Lingual Fine-Tuning
Fine-tuning pre-trained cross-lingual language models can transfer task-specific supervision from one language to the others.
Language Scaling for Universal Suggested Replies Model
Faced with increased compute requirements and low resources for language expansion, we build a single universal model for improving the quality and reducing run-time costs of our production system.
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
The first token-level task, Corrective Language Modeling, is to detect and correct tokens replaced by the auxiliary model, in order to better capture token-level semantics.
Pretrain Knowledge-Aware Language Models
Rather, we simply signal the existence of entities to the input of the transformer in pretraining, with an entity-extended tokenizer; and at the output, with an additional entity prediction task.
XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders
Multilingual machine translation enables a single model to translate between different languages.
Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point
In this paper, we explore the limits of Microsoft Floating Point (MSFP), a new class of datatypes developed for production cloud-scale inferencing on custom hardware.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts.
Ranked #16 on
Zero-Shot Cross-Lingual Transfer
on XTREME
Transformer-XH: Multi-Evidence Reasoning with eXtra Hop Attention
Transformers have achieved new heights modeling natural language as a sequence of text tokens.
Ranked #46 on
Question Answering
on HotpotQA
Generic Intent Representation in Web Search
This paper presents GEneric iNtent Encoder (GEN Encoder) which learns a distributed representation space for user intent in search.
An Axiomatic Approach to Regularizing Neural Ranking Models
The training of these models involve a search for appropriate parameter values based on large quantities of labeled examples.
Towards Language Agnostic Universal Representations
When a bilingual student learns to solve word problems in math, we expect the student to be able to solve these problem in both languages the student is fluent in, even if the math lessons were only taught in one language.
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
The size of the dataset and the fact that the questions are derived from real user search queries distinguishes MS MARCO from other well-known publicly available datasets for machine reading comprehension and question-answering.
