GPT-3 Explained | Papers With Code

GPT-3

Introduced by Brown et al. in Language Models are Few-Shot Learners

GPT-3 is an autoregressive transformer model with 175 billion parameters. It uses the same architecture/model as GPT-2, including the modified initialization, pre-normalization, and reversible tokenization, with the exception that GPT-3 uses alternating dense and locally banded sparse attention patterns in the layers of the transformer, similar to the Sparse Transformer.

Source: Language Models are Few-Shot Learners

Paper Code Snippet

Latest Papers

PAPER	DATE
One Model to Rule them All: Towards Zero-Shot Learning for Databases Benjamin Hilprecht • Carsten Binnig	2021-05-03
Unreasonable Effectiveness of Rule-Based Heuristics in Solving Russian SuperGLUE Tasks Tatyana Iazykova • Denis Kapelyushnik • Olga Bystrova • Andrey Kutuzov	2021-05-03
Entailment as Few-Shot Learner Sinong Wang • Han Fang • Madian Khabsa • Hanzi Mao • Hao Ma # 1 PARAPHRASE IDENTIFICATION ON QUORA QUESTION PAIRS	2021-04-29
PanGu-$α$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation Wei Zeng • Xiaozhe Ren • Teng Su • Hui Wang • Yi Liao • Zhiwei Wang • Xin Jiang • ZhenZhang Yang • Kaisheng Wang • Xiaoda Zhang • Chen Li • Ziyan Gong • Yifan Yao • Xinjing Huang • Jun Wang • Jianfeng Yu • Qi Guo • Yue Yu • Yan Zhang • Jin Wang • Hengtao Tao • Dasen Yan • Zexuan Yi • Fang Peng • Fangqing Jiang • Han Zhang • Lingfeng Deng • Yehong Zhang • Zhe Lin • Chao Zhang • Shaojie Zhang • Mingyue Guo • Shanzhi Gu • Gaojun Fan • YaoWei Wang • Xuefeng Jin • Qun Liu • Yonghong Tian # 1 READING COMPREHENSION (ZERO-SHOT) ON CMRC 2018	2021-04-26
GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation Kang Min Yoo • Dongju Park • Jaewook Kang • Sang-Woo Lee • Woomyeong Park	2021-04-18
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity Yao Lu • Max Bartolo • Alastair Moore • Sebastian Riedel • Pontus Stenetorp	2021-04-18
Natural Instructions: Benchmarking Generalization to New Tasks from Natural Language Instructions Swaroop Mishra • Daniel Khashabi • Chitta Baral • Hannaneh Hajishirzi	2021-04-18
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation Tianyu Liu • Yizhe Zhang • Chris Brockett • Yi Mao • Zhifang Sui • Weizhu Chen • Bill Dolan	2021-04-18
The Power of Scale for Parameter-Efficient Prompt Tuning \| Brian Lester • Rami Al-Rfou • Noah Constant	2021-04-18
An Adversarially-Learned Turing Test for Dialog Generation Models \| Xiang Gao • Yizhe Zhang • Michel Galley • Bill Dolan	2021-04-16
Text2App: A Framework for Creating Android Apps from Text Descriptions \| Masum Hasan • Kazi Sajeed Mehrab • Wasi Uddin Ahmad • Rifat Shahriyar	2021-04-16
Surface Form Competition: Why the Highest Probability Answer Isn't Always Right \| Ari Holtzman • Peter West • Vered Schwartz • Yejin Choi • Luke Zettlemoyer	2021-04-16
Meta-tuning Language Models to Answer Prompts Better Ruiqi Zhong • Kristy Lee • Zheng Zhang • Dan Klein	2021-04-10
Automatic Graph Partitioning for Very Large-scale Deep Learning Masahiro Tanaka • Kenjiro Taura • Toshihiro Hanawa • Kentaro Torisawa	2021-03-30
Detecting Hate Speech with GPT-3 \| Ke-Li Chiu • Rohan Alexander	2021-03-23
Calibrate Before Use: Improving Few-Shot Performance of Language Models \| Tony Z. Zhao • Eric Wallace • Shi Feng • Dan Klein • Sameer Singh	2021-02-19
TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models Zhuohan Li • Siyuan Zhuang • Shiyuan Guo • Danyang Zhuo • Hao Zhang • Dawn Song • Ion Stoica	2021-02-16
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm Laria Reynolds • Kyle McDonell	2021-02-15
Multiversal views on language models Laria Reynolds • Kyle McDonell	2021-02-12
PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers Chaoyang He • Shen Li • Mahdi Soltanolkotabi • Salman Avestimehr	2021-02-05
Understanding Emails and Drafting Responses -- An Approach Using GPT-3 Jonas Thiergart • Stefan Huber • Thomas Übellacker	2021-02-05
Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models Alex Tamkin • Miles Brundage • Jack Clark • Deep Ganguli	2021-02-04
"Is depression related to cannabis?": A knowledge-infused model for Entity and Relation Extraction with Limited Supervision Kaushik Roy • Usha Lokala • Vedant Khandelwal • Amit Sheth	2021-02-01
Persistent Anti-Muslim Bias in Large Language Models Abubakar Abid • Maheen Farooqi • James Zou	2021-01-14
How Multipurpose Are Language Models? Anonymous	2021-01-01
Making Pre-trained Language Models Better Few-shot Learners \| Tianyu Gao • Adam Fisch • Danqi Chen	2020-12-31
The Pile: An 800GB Dataset of Diverse Text for Language Modeling \| Leo Gao • Stella Biderman • Sid Black • Laurence Golding • Travis Hoppe • Charles Foster • Jason Phang • Horace He • Anish Thite • Noa Nabeshima • Shawn Presser • Connor Leahy	2020-12-31
Revisiting Linformer with a modified self-attention with linear complexity Madhusudan Verma	2020-12-16
Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment Julien Launay • Iacopo Poli • Kilian Müller • Gustave Pariente • Igor Carron • Laurent Daudet • Florent Krzakala • Sylvain Gigan	2020-12-11
CPM: A Large-scale Generative Chinese Pre-trained Language Model \| Zhengyan Zhang • Xu Han • Hao Zhou • Pei Ke • Yuxian Gu • Deming Ye • Yujia Qin • Yusheng Su • Haozhe Ji • Jian Guan • Fanchao Qi • Xiaozhi Wang • Yanan Zheng • Guoyang Zeng • Huanqi Cao • Shengqi Chen • Daixuan Li • Zhenbo Sun • Zhiyuan Liu • Minlie Huang • Wentao Han • Jie Tang • Juanzi Li • Xiaoyan Zhu • Maosong Sun	2020-12-01
Do Fine-tuned Commonsense Language Models Really Generalize? Mayank Kejriwal • Ke Shen	2020-11-18
COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs Jena D. Hwang • Chandra Bhagavatula • Ronan Le Bras • Jeff Da • Keisuke Sakaguchi • Antoine Bosselut • Yejin Choi	2020-10-12
Toward a Thermodynamics of Meaning \| Jonathan Scott Enderle	2020-09-24
It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners \| Timo Schick • Hinrich Schütze	2020-09-15
The Radicalization Risks of GPT-3 and Advanced Neural Language Models Kris McGuffie • Alex Newhouse	2020-09-15
Unit Test Case Generation with Transformers \| Michele Tufano • Dawn Drain • Alexey Svyatkovskiy • Shao Kun Deng • Neel Sundaresan	2020-09-11
Measuring Massive Multitask Language Understanding \| Dan Hendrycks • Collin Burns • Steven Basart • Andy Zou • Mantas Mazeika • Dawn Song • Jacob Steinhardt # 4 MULTI-TASK LEARNING ON HENDRYCKS TEST	2020-09-07
Discrete Word Embedding for Logical Natural Language Understanding Masataro Asai • Zilu Tang	2020-08-26
Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems Andrea Madotto • Zihan Liu • Zhaojiang Lin • Pascale Fung	2020-08-14
Language Models are Few-Shot Learners \| Tom B. Brown • Benjamin Mann • Nick Ryder • Melanie Subbiah • Jared Kaplan • Prafulla Dhariwal • Arvind Neelakantan • Pranav Shyam • Girish Sastry • Amanda Askell • Sandhini Agarwal • Ariel Herbert-Voss • Gretchen Krueger • Tom Henighan • Rewon Child • Aditya Ramesh • Daniel M. Ziegler • Jeffrey Wu • Clemens Winter • Christopher Hesse • Mark Chen • Eric Sigler • Mateusz Litwin • Scott Gray • Benjamin Chess • Jack Clark • Christopher Berner • Sam McCandlish • Alec Radford • Ilya Sutskever • Dario Amodei # 1 QUESTION ANSWERING ON PIQA	2020-05-28

Tasks

TASK	PAPERS	SHARE
Language Modelling	9	13.43%
Few-Shot Learning	7	10.45%
Natural Language Understanding	6	8.96%
Question Answering	5	7.46%
Zero-Shot Learning	3	4.48%
Natural Language Inference	3	4.48%
Text Classification	2	2.99%
Text Generation	2	2.99%
Multi-Task Learning	2	2.99%

Usage Over Time

This feature is experimental; we are continuously improving our matching algorithm.

Components

COMPONENT	TYPE	Add Remove
Adam	Stochastic Optimization
Attention Dropout	Regularization
BPE	Subword Segmentation
Dense Connections	Feedforward Networks
Dropout	Regularization
Fixed Factorized Attention	Attention Patterns
GELU	Activation Functions
Layer Normalization	Normalization
Linear Warmup With Cosine Annealing	Learning Rate Schedules
Multi-Head Attention	Attention Modules
Residual Connection	Skip Connections
Scaled Dot-Product Attention	Attention Mechanisms
Softmax	Output Functions
Strided Attention	Attention Patterns
Weight Decay	Regularization

Categories

Add Remove

Transformers

Language Models

Autoregressive Transformers