Search Results for author:
Found 23 papers, 8 papers with code
Scaling Laws of Synthetic Data for Language Models
Key findings from our extensive mathematical experiments on SynthLLM include: (1) SynthLLM generates synthetic data that reliably adheres to the rectified scaling law across various model sizes; (2) Performance improvements plateau near 300B tokens; and (3) Larger models approach optimal performance with fewer training tokens.
Dissecting In-Context Learning of Translations in GPTs
Most of the recent work in leveraging Large Language Models (LLMs) such as GPT-3 for Machine Translation (MT) has focused on selecting the few-shot samples for prompting.
Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness
In our comprehensive analysis, we show that MoE models with 2-bit expert weights can deliver better model performance than the dense model trained on the same dataset.
A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models
In this study, we propose a novel fine-tuning approach for LLMs that is specifically designed for the translation task, eliminating the need for the abundant parallel data that traditional translation models usually depend on.
Ranked #5 on
Machine Translation
on FLoRes-200
Task-Based MoE for Multitask Multilingual Machine Translation
Mixture-of-experts (MoE) architecture has been proven a powerful method for diverse tasks in training deep models in many applications.
FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs
Large Language Models (LLMs) have achieved state-of-the-art performance across various language tasks but pose challenges for practical deployment due to their substantial memory requirements.
Do GPTs Produce Less Literal Translations?
On the task of Machine Translation (MT), multiple works have investigated few-shot prompting mechanisms to elicit better translations from LLMs.
ResiDual: Transformer with Dual Residual Connections
In this paper, we propose ResiDual, a novel Transformer architecture with Pre-Post-LN (PPLN), which fuses the connections in Post-LN and Pre-LN together and inherits their advantages while avoids their limitations.
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
In this paper, we present a comprehensive evaluation of GPT models for machine translation, covering various aspects such as quality of different GPT models in comparison with state-of-the-art research and commercial systems, effect of prompting strategies, robustness towards domain shifts and document-level translation.
Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production
Mixture of Experts (MoE) models with conditional execution of sparsely activated layers have enabled training models with a much larger number of parameters.
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
Z-Code++ creates new state of the art on 9 out of 13 text summarization tasks across 5 languages.
Language Tokens: A Frustratingly Simple Approach Improves Zero-Shot Performance of Multilingual Translation
In a WMT evaluation campaign, From-English performance improves by 4. 17 and 2. 87 BLEU points, in the zero-shot setting, and when direct data is available for training, respectively.
Building Multilingual Machine Translation Systems That Serve Arbitrary X-Y Translations
Multilingual Neural Machine Translation (MNMT) enables one system to translate sentences from multiple source languages to multiple target languages, greatly reducing deployment costs compared with conventional bilingual systems.
Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers
Sparsely activated transformers, such as Mixture of Experts (MoE), have received great interest due to their outrageous scaling capability which enables dramatical increases in model size without significant increases in computational cost.
Ensembling of Distilled Models from Multi-task Teachers for Constrained Resource Language Pairs
This paper describes our submission to the constrained track of WMT21 shared news translation task.
Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task
This report describes Microsoft's machine translation systems for the WMT21 shared task on large-scale multilingual machine translation.
Scalable and Efficient MoE Training for Multitask Multilingual Models
By combining the efficient system and training methods, we are able to significantly scale up large multitask multilingual models for language generation which results in a great improvement in model accuracy.
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
While pretrained encoders have achieved success in various natural language understanding (NLU) tasks, there is a gap between these pretrained encoders and natural language generation (NLG).
XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders
Multilingual machine translation enables a single model to translate between different languages.
Score Combination for Improved Parallel Corpus Filtering for Low Resource Conditions
For the mBART finetuning setup, provided by the organizers, our method shows 7% and 5% relative improvement over baseline, in sacreBLEU score on the test set for Pashto and Khmer respectively.
FastFormers: Highly Efficient Transformer Models for Natural Language Understanding
In this paper, we present FastFormers, a set of recipes to achieve efficient inference-time performance for Transformer-based models on various NLU tasks.
Multi-task Learning for Multilingual Neural Machine Translation
In this work, we propose a multi-task learning (MTL) framework that jointly trains the model with the translation task on bitext data and two denoising tasks on the monolingual data.
Detecting Interrogative Utterances with Recurrent Neural Networks
In this paper, we explore different neural network architectures that can predict if a speaker of a given utterance is asking a question or making a statement.
