Search Results for author:
Found 38 papers, 16 papers with code
GenTool: Enhancing Tool Generalization in Language Models through Zero-to-One and Weak-to-Strong Simulation
Large Language Models (LLMs) can enhance their capabilities as AI assistants by integrating external tools, allowing them to access a wider range of information.
POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimization
In this work, we examine how the overgeneration of training data using advanced teacher models (e. g., GPT-4o), including responses to both general-purpose and toxic prompts, influences the safety and overrefusal balance of instruction-following language models.
Scaling Laws for Multilingual Language Models
We propose a novel scaling law for general-purpose decoder-only language models (LMs) trained on multilingual data, tackling the problem of balancing languages during multilingual pretraining.
Scaling Optimal LR Across Token Horizons
Secondly we demonstrate the the optimal LR follows a scaling law, and that the optimal LR for longer horizons can be accurately estimated from shorter horizons via such scaling laws.
WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback
As large language models (LLMs) continue to advance, aligning these models with human preferences has emerged as a critical challenge.
S2-Attention: Hardware-Aware Context Sharding Among Attention Heads
S2-Attention achieves wall-clock speedup of 8. 79X, 15. 87X, 25. 3X compared to the strong FlashAttention-2 baseline with strong downstream performance on-par with full attention and perfect retrieval performance at a 128k context length.
The Hitchhiker's Guide to Human Alignment with *PO
With the growing utilization of large language models (LLMs) across domains, alignment towards human preferences has become one of the most critical aspects of training models.
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
We introduce phi-3-mini, a 3. 8 billion parameter language model trained on 3. 3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3. 5 (e. g., phi-3-mini achieves 69% on MMLU and 8. 38 on MT-bench), despite being small enough to be deployed on a phone.
Ranked #5 on
MMR total
on MRR-Benchmark
(using extra training data)
Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models
Accurate and interpretable user satisfaction estimation (USE) is critical for understanding, evaluating, and continuously improving conversational systems.
GenSERP: Large Language Models for Whole Page Presentation
(2) An answer generation phase where the LLM populates the layouts with the retrieved content.
Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
This paper explores the effectiveness of model-generated signals in improving zero-shot generalization of text-to-text Transformers such as T5.
Language Is Not All You Need: Aligning Perception with Language Models
A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence.
TorchScale: Transformers at Scale
Large Transformers have achieved state-of-the-art performance across many tasks.
Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
In this paper, we elaborate upon recipes for building multilingual representation models that are not only competitive with existing state-of-the-art models but are also more parameter efficient, thereby promoting better adoption in resource-constrained scenarios and practical applications.
Foundation Transformers
A big convergence of model architectures across language, vision, speech, and multimodal is emerging.
On the Representation Collapse of Sparse Mixture of Experts
We also present a comprehensive analysis on the representation and routing behaviors of our models.
METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model.
Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators
We present a new framework AMOS that pretrains text encoders with an Adversarial learning curriculum via a Mixture Of Signals from multiple auxiliary generators.
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
Next, we detail the training process, the design of our training corpus, and our data curation techniques, which we believe is a key ingredient to the success of the model.
Ranked #35 on
Language Modelling
on LAMBADA
Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task
This report describes Microsoft's machine translation systems for the WMT21 shared task on large-scale multilingual machine translation.
Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
We find that many languages are under-represented in recent cross-lingual language models due to the limited vocabulary capacity.
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
In this paper, we introduce ELECTRA-style tasks to cross-lingual language model pre-training.
Ranked #1 on
Zero-Shot Cross-Lingual Transfer
on XTREME
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
While pretrained encoders have achieved success in various natural language understanding (NLU) tasks, there is a gap between these pretrained encoders and natural language generation (NLG).
Consistency Regularization for Cross-Lingual Fine-Tuning
Fine-tuning pre-trained cross-lingual language models can transfer task-specific supervision from one language to the others.
Language Scaling for Universal Suggested Replies Model
Faced with increased compute requirements and low resources for language expansion, we build a single universal model for improving the quality and reducing run-time costs of our production system.
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
The first token-level task, Corrective Language Modeling, is to detect and correct tokens replaced by the auxiliary model, in order to better capture token-level semantics.
Pretrain Knowledge-Aware Language Models
Rather, we simply signal the existence of entities to the input of the transformer in pretraining, with an entity-extended tokenizer; and at the output, with an additional entity prediction task.
XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders
Multilingual machine translation enables a single model to translate between different languages.
Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point
In this paper, we explore the limits of Microsoft Floating Point (MSFP), a new class of datatypes developed for production cloud-scale inferencing on custom hardware.
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts.
Ranked #16 on
Zero-Shot Cross-Lingual Transfer
on XTREME
Transformer-XH: Multi-Evidence Reasoning with eXtra Hop Attention
Transformers have achieved new heights modeling natural language as a sequence of text tokens.
Ranked #42 on
Question Answering
on HotpotQA
Generic Intent Representation in Web Search
This paper presents GEneric iNtent Encoder (GEN Encoder) which learns a distributed representation space for user intent in search.
An Axiomatic Approach to Regularizing Neural Ranking Models
The training of these models involve a search for appropriate parameter values based on large quantities of labeled examples.
Towards Language Agnostic Universal Representations
When a bilingual student learns to solve word problems in math, we expect the student to be able to solve these problem in both languages the student is fluent in, even if the math lessons were only taught in one language.
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
The size of the dataset and the fact that the questions are derived from real user search queries distinguishes MS MARCO from other well-known publicly available datasets for machine reading comprehension and question-answering.
