Search Results for author:
Found 23 papers, 10 papers with code
Scaling Laws for Multilingual Language Models
We propose a novel scaling law for general-purpose decoder-only language models (LMs) trained on multilingual data, tackling the problem of balancing languages during multilingual pretraining.
S2-Attention: Hardware-Aware Context Sharding Among Attention Heads
S2-Attention achieves wall-clock speedup of 8. 79X, 15. 87X, 25. 3X compared to the strong FlashAttention-2 baseline with strong downstream performance on-par with full attention and perfect retrieval performance at a 128k context length.
The Hitchhiker's Guide to Human Alignment with *PO
With the growing utilization of large language models (LLMs) across domains, alignment towards human preferences has become one of the most critical aspects of training models.
sPhinX: Sample Efficient Multilingual Instruction Fine-Tuning Through N-shot Guided Prompting
In order to address this, we introduce a novel recipe for creating a multilingual synthetic instruction tuning dataset, sPhinX, which is created by selectively translating instruction response pairs from English into 50 languages.
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
We introduce phi-3-mini, a 3. 8 billion parameter language model trained on 3. 3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3. 5 (e. g., phi-3-mini achieves 69% on MMLU and 8. 38 on MT-bench), despite being small enough to be deployed on a phone.
Ranked #5 on
MMR total
on MRR-Benchmark
(using extra training data)
A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia
We present a novel method to study grounding abilities using Fakepedia, a novel dataset of counterfactual texts constructed to clash with a model's internal parametric knowledge.
Language Is Not All You Need: Aligning Perception with Language Models
A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence.
TorchScale: Transformers at Scale
Large Transformers have achieved state-of-the-art performance across many tasks.
Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
In this paper, we elaborate upon recipes for building multilingual representation models that are not only competitive with existing state-of-the-art models but are also more parameter efficient, thereby promoting better adoption in resource-constrained scenarios and practical applications.
Language Model Decoding as Likelihood-Utility Alignment
Specifically, by analyzing the correlation between the likelihood and the utility of predictions across a diverse set of tasks, we provide empirical evidence supporting the proposed taxonomy and a set of principles to structure reasoning when choosing a decoding algorithm.
Foundation Transformers
A big convergence of model architectures across language, vision, speech, and multimodal is emerging.
On the Representation Collapse of Sparse Mixture of Experts
We also present a comprehensive analysis on the representation and routing behaviors of our models.
On Efficiently Acquiring Annotations for Multilingual Models
When tasked with supporting multiple languages for a given problem, two approaches have arisen: training a model for each language with the annotation budget divided equally among them, and training on a high-resource language followed by zero-shot transfer to the remaining languages.
Invariant Language Modeling
In particular, we adapt a game-theoretic formulation of IRM (IRM-games) to language models, where the invariance emerges from a specific training schedule in which all the environments compete to optimize their own environment-specific loss by updating subsets of the model in a round-robin fashion.
To Schedule or not to Schedule: Extracting Task Specific Temporal Entities and Associated Negation Constraints
State of the art research for date-time entity extraction from text is task agnostic.
ScopeIt: Scoping Task Relevant Sentences in Documents
Intelligent assistants like Cortana, Siri, Alexa, and Google Assistant are trained to parse information when the conversation is synchronous and short; however, for email-based conversational agents, the communication is asynchronous, and often contains information irrelevant to the assistant.
Weakly Supervised Attention Networks for Entity Recognition
The task of entity recognition has traditionally been modelled as a sequence labelling task.
Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spaces
We then propose Bilingual Lexicon Induction with Semi-Supervision (BLISS) --- a semi-supervised approach that relaxes the isometric assumption while leveraging both limited aligned bilingual lexicons and a larger set of unaligned word embeddings, as well as a novel hubness filtering technique.
Compression and Localization in Reinforcement Learning for ATARI Games
Deep neural networks have become commonplace in the domain of reinforcement learning, but are often expensive in terms of the number of parameters needed.
BLISS in Non-Isometric Embedding Spaces
We then propose Bilingual Lexicon Induction with Semi-Supervision (BLISS) --- a novel semi-supervised approach that relaxes the isometric assumption while leveraging both limited aligned bilingual lexicons and a larger set of unaligned word embeddings, as well as a novel hubness filtering technique.
Towards Understanding and Answering Multi-Sentence Recommendation Questions on Tourism
We introduce the first system towards the novel task of answering complex multisentence recommendation questions in the tourism domain.
