Search Results for author:
Found 7 papers, 3 papers with code
Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
Sparse Mixture-of-Experts (MoE) is a neural architecture design that can be utilized to add learnable parameters to Large Language Models (LLMs) without increasing inference cost.
Scaling Instruction-Finetuned Language Models
We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation).
Ranked #2 on
Multi-task Language Understanding
on BBH-alg
Dialog Inpainting: Turning Documents into Dialogs
Our approach takes the text of any document and transforms it into a two-person dialog between the writer and an imagined reader: we treat sentences from the article as utterances spoken by the writer, and then use a dialog inpainter to predict what the imagined reader asked or said in between each of the writer's utterances.
Mixture-of-Experts with Expert Choice Routing
Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens.
LaMDA: Language Models for Dialog Applications
We demonstrate that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding.
Feature Selection Facilitates Learning Mixtures of Discrete Product Distributions
Feature selection can facilitate the learning of mixtures of discrete random variables as they arise, e. g. in crowdsourcing tasks.
Stagewise Learning for Sparse Clustering of Discretely-Valued Data
The performance of EM in learning mixtures of product distributions often depends on the initialization.
