Search Results for author: Leo Gao

Found 7 papers, 4 papers with code

Datasheet for the Pile

1 code implementation13 Jan 2022 Stella Biderman, Kieran Bicheno, Leo Gao

This datasheet describes the Pile, a 825 GiB dataset of human-authored text compiled by EleutherAI for use in large-scale language modeling.

Language Modelling

Cut the CARP: Fishing for zero-shot story evaluation

no code implementations6 Oct 2021 Shahbuland Matiana, JR Smith, Ryan Teehan, Louis Castricato, Stella Biderman, Leo Gao, Spencer Frazier

Recent advances in large-scale language models (Raffel et al., 2019; Brown et al., 2020) have brought significant qualitative and quantitative improvements in machine-driven text generation.

Contrastive Learning Language Modelling +1

An Empirical Exploration in Quality Filtering of Text Data

no code implementations2 Sep 2021 Leo Gao

While conventional wisdom suggests that more aggressively filtering data from low-quality sources like Common Crawl always monotonically improves the quality of training data, we find that aggressive filtering can in fact lead to a decrease in model quality on a wide array of downstream tasks for a GPT-like language model.

Language Modelling

The Pile: An 800GB Dataset of Diverse Text for Language Modeling

11 code implementations31 Dec 2020 Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, Connor Leahy

Recent work has demonstrated that increased training dataset diversity improves general cross-domain knowledge and downstream generalization capability for large-scale language models.

Language Modelling

Collaborative Storytelling with Large-scale Neural Language Models

no code implementations20 Nov 2020 Eric Nichols, Leo Gao, Randy Gomez

We present a collaborative storytelling system which works with a human storyteller to create a story by generating new utterances based on the story so far.

Language Modelling

Cannot find the paper you are looking for? You can Submit a new open access paper.