Language Models


Introduced by Touvron et al. in LLaMA: Open and Efficient Foundation Language Models

LLaMA is a collection of foundation language models ranging from 7B to 65B parameters. It is based on the transformer architecture with various improvements that were subsequently proposed. The main difference with the original architecture are listed below.

  • RMSNorm normalizing function is used to improve the training stability, by normalizing the input of each transformer sub-layer, instead of normalizing the output.
  • The ReLU non-linearity is replaced by the SwiGLU activation function to improve performance.
  • Absolute positional embeddings are removed and instead rotary positional embeddings (RoPE) are added at each layer of the network.
Source: LLaMA: Open and Efficient Foundation Language Models


Paper Code Results Date Stars


Task Papers Share
Language Modelling 38 14.39%
Large Language Model 25 9.47%
Instruction Following 17 6.44%
Question Answering 10 3.79%
Text Generation 8 3.03%
Quantization 8 3.03%
Retrieval 6 2.27%
Code Generation 5 1.89%
Machine Translation 4 1.52%


Component Type
🤖 No Components Found You can add them if they exist; e.g. Mask R-CNN uses RoIAlign