TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Code Search	CodeXGLUE - AdvTest	CodeBERT	MRR	27.19	# 3
Clone Detection	CodeXGLUE - BigCloneBench	CodeBERT	F1	96.5	# 2
Code Repair	CodeXGLUE - Bugs2Fix	CodeBERT	BLEU (small)	77.42	# 1
Code Repair	CodeXGLUE - Bugs2Fix	CodeBERT	Accuracy (small)	16.4	# 2
Code Repair	CodeXGLUE - Bugs2Fix	CodeBERT	CodeBLEU (small)	75.58	# 1
Code Repair	CodeXGLUE - Bugs2Fix	CodeBERT	BLEU (medium)	91.07	# 1
Code Repair	CodeXGLUE - Bugs2Fix	CodeBERT	Accuracy (medium)	5.2	# 2
Code Repair	CodeXGLUE - Bugs2Fix	CodeBERT	CodeBLEU (medium)	87.52	# 1
Code Summarization	CodeXGLUE - CodeSearchNet	CodeBERT	Ruby	12.16	# 4
Code Summarization	CodeXGLUE - CodeSearchNet	CodeBERT	Javascript	14.9	# 4
Code Summarization	CodeXGLUE - CodeSearchNet	CodeBERT	Go	19.06	# 3
Code Summarization	CodeXGLUE - CodeSearchNet	CodeBERT	Python	18.07	# 5
Code Summarization	CodeXGLUE - CodeSearchNet	CodeBERT	Java	17.65	# 5
Code Summarization	CodeXGLUE - CodeSearchNet	CodeBERT	PHP	25.16	# 4
Code Translation	CodeXGLUE - CodeTrans	CodeBERT	BLEU (Java→C#)	79.92	# 2
Code Translation	CodeXGLUE - CodeTrans	CodeBERT	Accuracy (Java→C#)	59	# 2
Code Translation	CodeXGLUE - CodeTrans	CodeBERT	CodeBLEU (Java→C#)	85.1	# 1
Code Translation	CodeXGLUE - CodeTrans	CodeBERT	BLEU (C#→Java)	72.14	# 2
Code Translation	CodeXGLUE - CodeTrans	CodeBERT	Accuracy (C#→Java)	58	# 2
Code Translation	CodeXGLUE - CodeTrans	CodeBERT	CodeBLEU (C#→Java)	79.41	# 1
Text-to-Code Generation	CodeXGLUE - CONCODE	CodeGPT-adapted	EM	20.1	# 2
Text-to-Code Generation	CodeXGLUE - CONCODE	CodeGPT-adapted	BLEU	32.79	# 2
Text-to-Code Generation	CodeXGLUE - CONCODE	CodeGPT-adapted	CodeBLEU	27.74	# 2
Cloze Test	CodeXGLUE - CT-all	CodeBERT(MLM)	Ruby	80.17	# 1
Cloze Test	CodeXGLUE - CT-all	CodeBERT(MLM)	JS	81.77	# 1
Cloze Test	CodeXGLUE - CT-all	CodeBERT(MLM)	Go	83.31	# 1
Cloze Test	CodeXGLUE - CT-all	CodeBERT(MLM)	Python	87.21	# 1
Cloze Test	CodeXGLUE - CT-all	CodeBERT(MLM)	Java	80.63	# 1
Cloze Test	CodeXGLUE - CT-all	CodeBERT(MLM)	PHP	85.05	# 1
Cloze Test	CodeXGLUE - CT-maxmin	CodeBERT(MLM)	Ruby	86.84	# 1
Cloze Test	CodeXGLUE - CT-maxmin	CodeBERT(MLM)	JS	86.4	# 1
Cloze Test	CodeXGLUE - CT-maxmin	CodeBERT(MLM)	Go	90.79	# 1
Cloze Test	CodeXGLUE - CT-maxmin	CodeBERT(MLM)	Python	82.2	# 1
Cloze Test	CodeXGLUE - CT-maxmin	CodeBERT(MLM)	Java	90.46	# 1
Cloze Test	CodeXGLUE - CT-maxmin	CodeBERT(MLM)	PHP	88.21	# 1
Defect Detection	CodeXGLUE - Devign	CodeBERT	Accuracy	62.08	# 12
Code Completion	CodeXGLUE - Github Java Corpus	CodeGPT-adapted	Accuracy (token-level)	77.13	# 1
Code Completion	CodeXGLUE - Github Java Corpus	CodeGPT-adapted	EM (line-level)	26.43	# 3
Code Completion	CodeXGLUE - Github Java Corpus	CodeGPT-adapted	Edit Sim (line-level)	63.03	# 3
Document Translation	CodeXGLUE - Microsoft Docs	Pretrained Transformer	EN → DA	67.09	# 1
Document Translation	CodeXGLUE - Microsoft Docs	Pretrained Transformer	EN → LA	51.92	# 1
Document Translation	CodeXGLUE - Microsoft Docs	Pretrained Transformer	EN → NO	68.0	# 1
Document Translation	CodeXGLUE - Microsoft Docs	Pretrained Transformer	EN → ZH	70.6	# 1
Document Translation	CodeXGLUE - Microsoft Docs	Pretrained Transformer	DA → EN	67.02	# 1
Document Translation	CodeXGLUE - Microsoft Docs	Pretrained Transformer	LA → EN	68.3	# 1
Document Translation	CodeXGLUE - Microsoft Docs	Pretrained Transformer	NO → EN	71.84	# 1
Document Translation	CodeXGLUE - Microsoft Docs	Pretrained Transformer	ZH → EN	64.47	# 1
Clone Detection	CodeXGLUE - POJ-104	CodeBERT	MAP	84.29	# 1
Code Completion	CodeXGLUE - PY150	CodeGPT-adapted	Accuracy (token-level)	75.11	# 1
Code Completion	CodeXGLUE - PY150	CodeGPT-adapted	EM (line-level)	39.65	# 3
Code Completion	CodeXGLUE - PY150	CodeGPT-adapted	Edit Sim (line-level)	69.84	# 3
Code Search	CodeXGLUE - WebQueryTest	CodeBERT	F1	58.95	# 1
Code Search	CodeXGLUE - WebQueryTest	CodeBERT	Accuracy	47.8	# 1

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codexglue-a-machine-learning-benchmark/cloze-test-on-codexglue-ct-all)](https://paperswithcode.com/sota/cloze-test-on-codexglue-ct-all?p=codexglue-a-machine-learning-benchmark)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codexglue-a-machine-learning-benchmark/cloze-test-on-codexglue-ct-maxmin)](https://paperswithcode.com/sota/cloze-test-on-codexglue-ct-maxmin?p=codexglue-a-machine-learning-benchmark)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codexglue-a-machine-learning-benchmark/code-completion-on-codexglue-github-java)](https://paperswithcode.com/sota/code-completion-on-codexglue-github-java?p=codexglue-a-machine-learning-benchmark)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codexglue-a-machine-learning-benchmark/document-translation-on-codexglue-microsoft)](https://paperswithcode.com/sota/document-translation-on-codexglue-microsoft?p=codexglue-a-machine-learning-benchmark)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codexglue-a-machine-learning-benchmark/clone-detection-on-codexglue-poj-104)](https://paperswithcode.com/sota/clone-detection-on-codexglue-poj-104?p=codexglue-a-machine-learning-benchmark)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codexglue-a-machine-learning-benchmark/code-completion-on-codexglue-py150)](https://paperswithcode.com/sota/code-completion-on-codexglue-py150?p=codexglue-a-machine-learning-benchmark)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codexglue-a-machine-learning-benchmark/code-search-on-codexglue-webquerytest)](https://paperswithcode.com/sota/code-search-on-codexglue-webquerytest?p=codexglue-a-machine-learning-benchmark)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codexglue-a-machine-learning-benchmark/clone-detection-on-codexglue-bigclonebench)](https://paperswithcode.com/sota/clone-detection-on-codexglue-bigclonebench?p=codexglue-a-machine-learning-benchmark)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codexglue-a-machine-learning-benchmark/code-repair-on-codexglue-bugs2fix)](https://paperswithcode.com/sota/code-repair-on-codexglue-bugs2fix?p=codexglue-a-machine-learning-benchmark)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codexglue-a-machine-learning-benchmark/code-translation-on-codexglue-codetrans)](https://paperswithcode.com/sota/code-translation-on-codexglue-codetrans?p=codexglue-a-machine-learning-benchmark)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codexglue-a-machine-learning-benchmark/text-to-code-generation-on-codexglue-concode)](https://paperswithcode.com/sota/text-to-code-generation-on-codexglue-concode?p=codexglue-a-machine-learning-benchmark)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codexglue-a-machine-learning-benchmark/code-search-on-codexglue-advtest)](https://paperswithcode.com/sota/code-search-on-codexglue-advtest?p=codexglue-a-machine-learning-benchmark)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codexglue-a-machine-learning-benchmark/code-summarization-on-codexglue-codesearchnet)](https://paperswithcode.com/sota/code-summarization-on-codexglue-codesearchnet?p=codexglue-a-machine-learning-benchmark)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codexglue-a-machine-learning-benchmark/defect-detection-on-codexglue-devign)](https://paperswithcode.com/sota/defect-detection-on-codexglue-devign?p=codexglue-a-machine-learning-benchmark)`

CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation

9 Feb 2021 · Shuai Lu, Daya Guo, Shuo Ren, JunJie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, Shujie Liu ·

Benchmark datasets have a significant impact on accelerating research in programming language tasks. In this paper, we introduce CodeXGLUE, a benchmark dataset to foster machine learning research for program understanding and generation. CodeXGLUE includes a collection of 10 tasks across 14 datasets and a platform for model evaluation and comparison. CodeXGLUE also features three baseline systems, including the BERT-style, GPT-style, and Encoder-Decoder models, to make it easy for researchers to use the platform. The availability of such data and baselines can help the development and validation of new methods that can be applied to various program understanding and generation problems.

PDF Abstract

Code

Add Remove Mark official

microsoft/CodeXGLUE official

1,514

facebookresearch/CodeGen

704

sberbank-ai/fusion_brain_aij2021

↳ Quickstart in

Colab

yueyuel/programgen-lms-reliability

Tasks

Add Remove

BIG-bench Machine Learning

Clone Detection

Cloze Test

Code Completion

Code Generation

Code Repair

Code Search

Code Summarization

Code Translation

Decoder

Defect Detection

Document Translation

Text-to-Code Generation

Datasets

Introduced in the Paper:

CodeXGLUE

Used in the Paper:

ImageNet

GLUE CodeSearchNet CONCODE XGLUE

Results from the Paper

Edit

Ranked #1 on Cloze Test on CodeXGLUE - CT-maxmin

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Code Search	CodeXGLUE - AdvTest	CodeBERT	MRR	27.19	# 3	Compare
Clone Detection	CodeXGLUE - BigCloneBench	CodeBERT	F1	96.5	# 2	Compare
Code Repair	CodeXGLUE - Bugs2Fix	CodeBERT	BLEU (small)	77.42	# 1	Compare
			Accuracy (small)	16.4	# 2	Compare
			CodeBLEU (small)	75.58	# 1	Compare
			BLEU (medium)	91.07	# 1	Compare
			Accuracy (medium)	5.2	# 2	Compare
			CodeBLEU (medium)	87.52	# 1	Compare
Code Summarization	CodeXGLUE - CodeSearchNet	CodeBERT	Ruby	12.16	# 4	Compare
			Javascript	14.9	# 4	Compare
			Go	19.06	# 3	Compare
			Python	18.07	# 5	Compare
			Java	17.65	# 5	Compare
			PHP	25.16	# 4	Compare
Code Translation	CodeXGLUE - CodeTrans	CodeBERT	BLEU (Java→C#)	79.92	# 2	Compare
			Accuracy (Java→C#)	59	# 2	Compare
			CodeBLEU (Java→C#)	85.1	# 1	Compare
			BLEU (C#→Java)	72.14	# 2	Compare
			Accuracy (C#→Java)	58	# 2	Compare
			CodeBLEU (C#→Java)	79.41	# 1	Compare
Text-to-Code Generation	CodeXGLUE - CONCODE	CodeGPT-adapted	EM	20.1	# 2	Compare
			BLEU	32.79	# 2	Compare
			CodeBLEU	27.74	# 2	Compare
Cloze Test	CodeXGLUE - CT-all	CodeBERT(MLM)	Ruby	80.17	# 1	Compare
			JS	81.77	# 1	Compare
			Go	83.31	# 1	Compare
			Python	87.21	# 1	Compare
			Java	80.63	# 1	Compare
			PHP	85.05	# 1	Compare
Cloze Test	CodeXGLUE - CT-maxmin	CodeBERT(MLM)	Ruby	86.84	# 1	Compare
			JS	86.4	# 1	Compare
			Go	90.79	# 1	Compare
			Python	82.2	# 1	Compare
			Java	90.46	# 1	Compare
			PHP	88.21	# 1	Compare
Defect Detection	CodeXGLUE - Devign	CodeBERT	Accuracy	62.08	# 12	Compare
Code Completion	CodeXGLUE - Github Java Corpus	CodeGPT-adapted	Accuracy (token-level)	77.13	# 1	Compare
			EM (line-level)	26.43	# 3	Compare
			Edit Sim (line-level)	63.03	# 3	Compare
Document Translation	CodeXGLUE - Microsoft Docs	Pretrained Transformer	EN → DA	67.09	# 1	Compare
			EN → LA	51.92	# 1	Compare
			EN → NO	68.0	# 1	Compare
			EN → ZH	70.6	# 1	Compare
			DA → EN	67.02	# 1	Compare
			LA → EN	68.3	# 1	Compare
			NO → EN	71.84	# 1	Compare
			ZH → EN	64.47	# 1	Compare
Clone Detection	CodeXGLUE - POJ-104	CodeBERT	MAP	84.29	# 1	Compare
Code Completion	CodeXGLUE - PY150	CodeGPT-adapted	Accuracy (token-level)	75.11	# 1	Compare
			EM (line-level)	39.65	# 3	Compare
			Edit Sim (line-level)	69.84	# 3	Compare
Code Search	CodeXGLUE - WebQueryTest	CodeBERT	F1	58.95	# 1	Compare
Code Search	CodeXGLUE - WebQueryTest	CodeBERT	Accuracy	47.8	# 1	Compare

Methods

Add Remove

No methods listed for this paper. Add relevant methods here

Edit Social Preview

CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove