GPT-3

Introduced by Brown et al. in Language Models are Few-Shot Learners

GPT-3 is an autoregressive transformer model with 175 billion parameters. It uses the same architecture/model as GPT-2, including the modified initialization, pre-normalization, and reversible tokenization, with the exception that GPT-3 uses alternating dense and locally banded sparse attention patterns in the layers of the transformer, similar to the Sparse Transformer.

Source: Language Models are Few-Shot Learners

Read Paper See Code

Papers

Paper	Code	Results	Date	Stars

Tasks

Task	Papers	Share
Language Modelling	74	9.67%
Question Answering	49	6.41%
Large Language Model	46	6.01%
Retrieval	32	4.18%
In-Context Learning	32	4.18%
Code Generation	28	3.66%
Prompt Engineering	25	3.27%
Sentence	23	3.01%
Text Generation	19	2.48%

Usage Over Time

This feature is experimental; we are continuously improving our matching algorithm.

Components

Component	Type	Add Remove
Adam	Stochastic Optimization
Attention Dropout	Regularization
BPE	Subword Segmentation
Dense Connections	Feedforward Networks
Dropout	Regularization
Fixed Factorized Attention	Attention Patterns
GELU	Activation Functions
Layer Normalization	Normalization
Linear Warmup With Cosine Annealing	Learning Rate Schedules
Multi-Head Attention	Attention Modules
Residual Connection	Skip Connections
Scaled Dot-Product Attention	Attention Mechanisms
Softmax	Output Functions
Strided Attention	Attention Patterns
Weight Decay	Regularization

Categories

Add Remove

Transformers

Language Models

Autoregressive Transformers