TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
					# 1
Question Answering	BoolQ	LLaMA 2 70B (zero-shot)	Accuracy	85	# 12
Question Answering	BoolQ	LLaMA 2 13B (zero-shot)	Accuracy	81.7	# 18
Question Answering	BoolQ	LLaMA 2 7B (zero-shot)	Accuracy	77.4	# 22
Question Answering	BoolQ	LLaMA 2 34B (zero-shot)	Accuracy	83.7	# 14
Arithmetic Reasoning	GSM8K	LLaMA 2 70B (on-shot)	Accuracy	56.8	# 56
Arithmetic Reasoning	GSM8K	LLaMA 2 70B (on-shot)	Parameters (Billion)	70	# 35
Sentence Completion	HellaSwag	LLaMA 2 7B (zero-shot)	Accuracy	77.2	# 27
Sentence Completion	HellaSwag	LLaMA 2 70B (zero-shot)	Accuracy	85.3	# 9
Sentence Completion	HellaSwag	LLaMA 2 13B (zero-shot)	Accuracy	80.7	# 21
Sentence Completion	HellaSwag	LLaMA 2 34B (zero-shot)	Accuracy	83.3	# 15
Code Generation	HumanEval	LLaMA 2 (zero-shot)	Pass@1	29.9	# 33
Math Word Problem Solving	MAWPS	LLaMA 2-Chat	Accuracy (%)	82.4	# 14
Multi-task Language Understanding	MMLU	LLaMA 2 34B (few-shot, k=5)	Average (%)	62.6	# 24
Multi-task Language Understanding	MMLU	LLaMA 2 13B (few-shot, k=5)	Average (%)	54.8	# 32
Multi-task Language Understanding	MMLU	LLaMA 2 7B (few-shot, k=5)	Average (%)	45.3	# 42
Multi-task Language Understanding	MMLU	LLaMA 2 70B (few-shot, k=5)	Average (%)	68.9	# 18
Multi-task Language Understanding	MMLU	LLaMA 2 70B (few-shot, k=5)	Parameters (Billions)	70	# 33
Multiple Choice Question Answering (MCQA)	MMLU (Professional medicine)	Llama2-7B-chat	Accuracy	40.07	# 6
Multiple Choice Question Answering (MCQA)	MMLU (Professional medicine)	Llama2-7B	Accuracy	43.38	# 5
Question Answering	Natural Questions	LLaMA 2 70B (one-shot)	EM	33.0	# 22
Question Answering	PIQA	LLaMA 2 7B (zero-shot)	Accuracy	78.8	# 21
Question Answering	PIQA	LLaMA 2 34B (zero-shot)	Accuracy	81.9	# 9
Question Answering	PIQA	LLaMA 2 70B (zero-shot)	Accuracy	82.8	# 4
Question Answering	PIQA	LLaMA 2 13B (zero-shot)	Accuracy	80.5	# 15
Question Answering	PubChemQA	Llama2-7B-chat	BLEU-2	0.075	# 2
Question Answering	PubChemQA	Llama2-7B-chat	BLEU-4	0.009	# 2
Question Answering	PubChemQA	Llama2-7B-chat	ROUGE-1	0.184	# 2
Question Answering	PubChemQA	Llama2-7B-chat	ROUGE-2	0.043	# 2
Question Answering	PubChemQA	Llama2-7B-chat	ROUGE-L	0.142	# 2
Question Answering	PubChemQA	Llama2-7B-chat	MEATOR	0.149	# 2
Math Word Problem Solving	SVAMP	LLaMA 2-Chat	Execution Accuracy	69.2	# 4
Question Answering	TriviaQA	LLaMA 2 70B (one-shot)	EM	85	# 2
Question Answering	UniProtQA	Llama2-7B-chat	BLEU-2	0.019	# 2
Question Answering	UniProtQA	Llama2-7B-chat	BLEU-4	0.002	# 2
Question Answering	UniProtQA	Llama2-7B-chat	ROUGE-1	0.103	# 2
Question Answering	UniProtQA	Llama2-7B-chat	ROUGE-2	0.060	# 2
Question Answering	UniProtQA	Llama2-7B-chat	ROUGE-L	0.009	# 2
Question Answering	UniProtQA	Llama2-7B-chat	MEATOR	0.052	# 2

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/llama-2-open-foundation-and-fine-tuned-chat/on-1)](https://paperswithcode.com/sota/on-1?p=llama-2-open-foundation-and-fine-tuned-chat)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/llama-2-open-foundation-and-fine-tuned-chat/question-answering-on-pubchemqa)](https://paperswithcode.com/sota/question-answering-on-pubchemqa?p=llama-2-open-foundation-and-fine-tuned-chat)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/llama-2-open-foundation-and-fine-tuned-chat/question-answering-on-triviaqa)](https://paperswithcode.com/sota/question-answering-on-triviaqa?p=llama-2-open-foundation-and-fine-tuned-chat)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/llama-2-open-foundation-and-fine-tuned-chat/question-answering-on-uniprotqa)](https://paperswithcode.com/sota/question-answering-on-uniprotqa?p=llama-2-open-foundation-and-fine-tuned-chat)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/llama-2-open-foundation-and-fine-tuned-chat/question-answering-on-piqa)](https://paperswithcode.com/sota/question-answering-on-piqa?p=llama-2-open-foundation-and-fine-tuned-chat)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/llama-2-open-foundation-and-fine-tuned-chat/math-word-problem-solving-on-svamp)](https://paperswithcode.com/sota/math-word-problem-solving-on-svamp?p=llama-2-open-foundation-and-fine-tuned-chat)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/llama-2-open-foundation-and-fine-tuned-chat/multiple-choice-question-answering-mcqa-on-25)](https://paperswithcode.com/sota/multiple-choice-question-answering-mcqa-on-25?p=llama-2-open-foundation-and-fine-tuned-chat)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/llama-2-open-foundation-and-fine-tuned-chat/sentence-completion-on-hellaswag)](https://paperswithcode.com/sota/sentence-completion-on-hellaswag?p=llama-2-open-foundation-and-fine-tuned-chat)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/llama-2-open-foundation-and-fine-tuned-chat/question-answering-on-boolq)](https://paperswithcode.com/sota/question-answering-on-boolq?p=llama-2-open-foundation-and-fine-tuned-chat)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/llama-2-open-foundation-and-fine-tuned-chat/math-word-problem-solving-on-mawps)](https://paperswithcode.com/sota/math-word-problem-solving-on-mawps?p=llama-2-open-foundation-and-fine-tuned-chat)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/llama-2-open-foundation-and-fine-tuned-chat/multi-task-language-understanding-on-mmlu)](https://paperswithcode.com/sota/multi-task-language-understanding-on-mmlu?p=llama-2-open-foundation-and-fine-tuned-chat)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/llama-2-open-foundation-and-fine-tuned-chat/question-answering-on-natural-questions)](https://paperswithcode.com/sota/question-answering-on-natural-questions?p=llama-2-open-foundation-and-fine-tuned-chat)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/llama-2-open-foundation-and-fine-tuned-chat/code-generation-on-humaneval)](https://paperswithcode.com/sota/code-generation-on-humaneval?p=llama-2-open-foundation-and-fine-tuned-chat)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/llama-2-open-foundation-and-fine-tuned-chat/arithmetic-reasoning-on-gsm8k)](https://paperswithcode.com/sota/arithmetic-reasoning-on-gsm8k?p=llama-2-open-foundation-and-fine-tuned-chat)`

Llama 2: Open Foundation and Fine-Tuned Chat Models

18 Jul 2023 · Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, Thomas Scialom ·

In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for dialogue use cases. Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closed-source models. We provide a detailed description of our approach to fine-tuning and safety improvements of Llama 2-Chat in order to enable the community to build on our work and contribute to the responsible development of LLMs.

PDF Abstract

Code

Add Remove Mark official

facebookresearch/llama official

46,767

flagalpha/llama2-chinese

7,340

Lightning-AI/lit-gpt

3,992

IBM/Dromedary

1,027

xverse-ai/xverse-13b

554

See all 11 implementations

Tasks

Add Remove

Arithmetic Reasoning

Code Generation

Math Word Problem Solving

Multiple Choice Question Answering (MCQA)

Multi-task Language Understanding

Question Answering

Sentence Completion

Datasets

Natural Questions

SuperGLUE

TriviaQA

GSM8K

MMLU

BoolQ

CommonsenseQA

HumanEval

HellaSwag

PIQA

MATH

ARC

SVAMP

TruthfulQA MAWPS

ToxiGen PubChemQA UniProtQA

Results from the Paper

Edit

Ranked #1 on on

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
					# 1	Compare
Question Answering	BoolQ	LLaMA 2 70B (zero-shot)	Accuracy	85	# 12	Compare
Question Answering	BoolQ	LLaMA 2 13B (zero-shot)	Accuracy	81.7	# 18	Compare
Question Answering	BoolQ	LLaMA 2 7B (zero-shot)	Accuracy	77.4	# 22	Compare
Question Answering	BoolQ	LLaMA 2 34B (zero-shot)	Accuracy	83.7	# 14	Compare
Arithmetic Reasoning	GSM8K	LLaMA 2 70B (on-shot)	Accuracy	56.8	# 56	Compare
Arithmetic Reasoning	GSM8K	LLaMA 2 70B (on-shot)	Parameters (Billion)	70	# 35	Compare
Sentence Completion	HellaSwag	LLaMA 2 7B (zero-shot)	Accuracy	77.2	# 27	Compare
Sentence Completion	HellaSwag	LLaMA 2 70B (zero-shot)	Accuracy	85.3	# 9	Compare
Sentence Completion	HellaSwag	LLaMA 2 13B (zero-shot)	Accuracy	80.7	# 21	Compare
Sentence Completion	HellaSwag	LLaMA 2 34B (zero-shot)	Accuracy	83.3	# 15	Compare
Code Generation	HumanEval	LLaMA 2 (zero-shot)	Pass@1	29.9	# 33	Compare
Math Word Problem Solving	MAWPS	LLaMA 2-Chat	Accuracy (%)	82.4	# 14	Compare
Multi-task Language Understanding	MMLU	LLaMA 2 34B (few-shot, k=5)	Average (%)	62.6	# 24	Compare
Multi-task Language Understanding	MMLU	LLaMA 2 13B (few-shot, k=5)	Average (%)	54.8	# 32	Compare
Multi-task Language Understanding	MMLU	LLaMA 2 7B (few-shot, k=5)	Average (%)	45.3	# 42	Compare
Multi-task Language Understanding	MMLU	LLaMA 2 70B (few-shot, k=5)	Average (%)	68.9	# 18	Compare
Multi-task Language Understanding	MMLU	LLaMA 2 70B (few-shot, k=5)	Parameters (Billions)	70	# 33	Compare
Multiple Choice Question Answering (MCQA)	MMLU (Professional medicine)	Llama2-7B-chat	Accuracy	40.07	# 6	Compare
Multiple Choice Question Answering (MCQA)	MMLU (Professional medicine)	Llama2-7B	Accuracy	43.38	# 5	Compare
Question Answering	Natural Questions	LLaMA 2 70B (one-shot)	EM	33.0	# 22	Compare
Question Answering	PIQA	LLaMA 2 7B (zero-shot)	Accuracy	78.8	# 21	Compare
Question Answering	PIQA	LLaMA 2 34B (zero-shot)	Accuracy	81.9	# 9	Compare
Question Answering	PIQA	LLaMA 2 70B (zero-shot)	Accuracy	82.8	# 4	Compare
Question Answering	PIQA	LLaMA 2 13B (zero-shot)	Accuracy	80.5	# 15	Compare
Question Answering	PubChemQA	Llama2-7B-chat	BLEU-2	0.075	# 2	Compare
			BLEU-4	0.009	# 2	Compare
			ROUGE-1	0.184	# 2	Compare
			ROUGE-2	0.043	# 2	Compare
			ROUGE-L	0.142	# 2	Compare
			MEATOR	0.149	# 2	Compare
Math Word Problem Solving	SVAMP	LLaMA 2-Chat	Execution Accuracy	69.2	# 4	Compare
Question Answering	TriviaQA	LLaMA 2 70B (one-shot)	EM	85	# 2	Compare
Question Answering	UniProtQA	Llama2-7B-chat	BLEU-2	0.019	# 2	Compare
			BLEU-4	0.002	# 2	Compare
			ROUGE-1	0.103	# 2	Compare
			ROUGE-2	0.060	# 2	Compare
			ROUGE-L	0.009	# 2	Compare
			MEATOR	0.052	# 2	Compare

Methods

Add Remove

Absolute Position Encodings • AdamW • BPE • Dense Connections • Dropout • Entropy Regularization • Feedforward Network • Grouped-query attention • Label Smoothing • LLaMA • PPO • Residual Connection • RMSNorm • Rotary Embeddings • Scaled Dot-Product Attention • Softmax • SwiGLU • Transformer

Edit Social Preview

Llama 2: Open Foundation and Fine-Tuned Chat Models

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove