The Falcon Series of Open Language Models

no code implementations • 28 Nov 2023 • Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, Guilherme Penedo

We report detailed evaluations, as well as a deep dive into the methods and custom tooling employed to pretrain Falcon.

Multi-task Language Understanding Sentence Completion

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only

1 code implementation • 1 Jun 2023 • Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, Julien Launay

Large language models are commonly trained on a mixture of filtered web data and curated high-quality corpora, such as social media conversations, books, or technical papers.

Zero-shot Generalization

PAGnol: An Extra-Large French Generative Model

no code implementations • LREC 2022 • Julien Launay, E. L. Tommasone, Baptiste Pannier, François Boniface, Amélie Chatelain, Alessandro Cappelli, Iacopo Poli, Djamé Seddah

We fit a scaling law for compute for the French language, and compare it with its English counterpart.

Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning

no code implementations • ACL 2020 • Alexandre Tamborrino, Nicola Pellicano, Baptiste Pannier, Pascal Voitot, Louise Naudin

Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks.

Language Modelling Masked Language Modeling