TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Code Documentation Generation	CodeSearchNet	seq2seq	Smoothed BLEU-4	13.36	# 7
Code Documentation Generation	CodeSearchNet	Transformer	Smoothed BLEU-4	14.31	# 6
Code Search	CodeSearchNet	CodeBERT	Overall	76.0	# 6
Code Search	CodeSearchNet	CodeBERT	Go	69.3	# 6
Code Search	CodeSearchNet	CodeBERT	Ruby	70.6	# 6
Code Search	CodeSearchNet	CodeBERT	Python	84.0	# 4
Code Search	CodeSearchNet	CodeBERT	Java	86.8	# 3
Code Search	CodeSearchNet	CodeBERT	JS	74.8	# 3
Code Search	CodeSearchNet	CodeBERT	PHP	70.6	# 4
Code Documentation Generation	CodeSearchNet	CodeBERT (MLM+RTD)	Smoothed BLEU-4	15.99	# 1
Code Documentation Generation	CodeSearchNet	RoBERTa	Smoothed BLEU-4	14.52	# 5
Code Documentation Generation	CodeSearchNet	CodeBERT (MLM)	Smoothed BLEU-4	15.55	# 2
Code Documentation Generation	CodeSearchNet	pre-train w/ code only	Smoothed BLEU-4	15.15	# 3
Code Documentation Generation	CodeSearchNet	CodeBERT (RTD)	Smoothed BLEU-4	15.03	# 4
Code Documentation Generation	CodeSearchNet - Go	CodeBERT (RTD)	Smoothed BLEU-4	26.02	# 5
Code Documentation Generation	CodeSearchNet - Go	RoBERTa	Smoothed BLEU-4	26.09	# 4
Code Documentation Generation	CodeSearchNet - Go	CodeBERT (MLM)	Smoothed BLEU-4	26.79	# 1
Code Documentation Generation	CodeSearchNet - Go	seq2seq	Smoothed BLEU-4	23.48	# 6
Code Documentation Generation	CodeSearchNet - Go	pre-train w/ code only	Smoothed BLEU-4	26.39	# 3
Code Documentation Generation	CodeSearchNet - Go	CodeBERT (MLM+RTD)	Smoothed BLEU-4	26.66	# 2
Code Documentation Generation	CodeSearchNet - Java	Transformer	Smoothed BLEU-4	12.57	# 7
Code Documentation Generation	CodeSearchNet - Java	CodeBERT (MLM+RTD)	Smoothed BLEU-4	14.56	# 2
Code Documentation Generation	CodeSearchNet - Java	CodeBERT (MLM)	Smoothed BLEU-4	13.59	# 3
Code Documentation Generation	CodeSearchNet - Java	CodeBERT (RTD)	Smoothed BLEU-4	12.72	# 6
Code Documentation Generation	CodeSearchNet - Java	pre-train w/ code only	Smoothed BLEU-4	13.07	# 5
Code Documentation Generation	CodeSearchNet - Java	RoBERTa	Smoothed BLEU-4	13.2	# 4
Code Documentation Generation	CodeSearchNet - Java	seq2seq	Smoothed BLEU-4	11.42	# 8
Code Documentation Generation	CodeSearchNet - JavaScript	CodeBERT (MLM)	Smoothed BLEU-4	8.51	# 5
Code Documentation Generation	CodeSearchNet - JavaScript	CodeBERT (RTD)	Smoothed BLEU-4	8.73	# 4
Code Documentation Generation	CodeSearchNet - JavaScript	Transformer	Smoothed BLEU-4	25.61	# 1
Code Documentation Generation	CodeSearchNet - JavaScript	CodeBERT (MLM+RTD)	Smoothed BLEU-4	9.54	# 3
Code Documentation Generation	CodeSearchNet - JavaScript	RoBERTa	Smoothed BLEU-4	5.72	# 8
Code Documentation Generation	CodeSearchNet - JavaScript	pre-train w/ code only	Smoothed BLEU-4	8.3	# 6
Code Documentation Generation	CodeSearchNet - JavaScript	seq2seq	Smoothed BLEU-4	6.88	# 7
Code Documentation Generation	CodeSearchNet - Php	Transformer	Smoothed BLEU-4	18.25	# 8
Code Documentation Generation	CodeSearchNet - Php	CodeBERT (MLM)	Smoothed BLEU-4	21	# 3
Code Documentation Generation	CodeSearchNet - Php	pre-train w/ code only	Smoothed BLEU-4	20.71	# 4
Code Documentation Generation	CodeSearchNet - Php	RoBERTa	Smoothed BLEU-4	19.9	# 6
Code Documentation Generation	CodeSearchNet - Php	CodeBERT (MLM+RTD)	Smoothed BLEU-4	21.32	# 2
Code Documentation Generation	CodeSearchNet - Php	seq2seq	Smoothed BLEU-4	18.4	# 7
Code Documentation Generation	CodeSearchNet - Php	CodeBERT (RTD)	Smoothed BLEU-4	20.25	# 5
Code Documentation Generation	CodeSearchNet - Python	pre-train w/ code only	Smoothed BLEU-4	15.12	# 4
Code Documentation Generation	CodeSearchNet - Python	CodeBERT (MLM+RTD)	Smoothed BLEU-4	15.41	# 3
Code Documentation Generation	CodeSearchNet - Python	Transformer	Smoothed BLEU-4	13.44	# 6
Code Documentation Generation	CodeSearchNet - Python	seq2seq	Smoothed BLEU-4	13.04	# 7
Code Documentation Generation	CodeSearchNet - Python	CodeBERT (MLM)	Smoothed BLEU-4	15.48	# 2
Code Documentation Generation	CodeSearchNet - Python	RoBERTa	Smoothed BLEU-4	14.92	# 5
Code Documentation Generation	CodeSearchNet - Ruby	Transformer	Smoothed BLEU-4	7.87	# 4
Code Documentation Generation	CodeSearchNet - Ruby	pre-train w/ code only	Smoothed BLEU-4	7.36	# 5
Code Documentation Generation	CodeSearchNet - Ruby	CodeBERT (MLM)	Smoothed BLEU-4	7.95	# 3
Code Documentation Generation	CodeSearchNet - Ruby	seq2seq	Smoothed BLEU-4	6.96	# 7
Code Documentation Generation	CodeSearchNet - Ruby	CodeBERT (MLM+RTD)	Smoothed BLEU-4	8.46	# 2
Code Documentation Generation	CodeSearchNet - Ruby	RoBERTa	Smoothed BLEU-4	7.26	# 6
Type prediction	ManyTypes4TypeScript	CodeBERT	Average Accuracy	61.72	# 4
Type prediction	ManyTypes4TypeScript	CodeBERT	Average Precision	59.34	# 2
Type prediction	ManyTypes4TypeScript	CodeBERT	Average Recall	59.80	# 2
Type prediction	ManyTypes4TypeScript	CodeBERT	Average F1	59.57	# 2

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codebert-a-pre-trained-model-for-programming/code-documentation-generation-on-6)](https://paperswithcode.com/sota/code-documentation-generation-on-6?p=codebert-a-pre-trained-model-for-programming)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codebert-a-pre-trained-model-for-programming/code-documentation-generation-on-2)](https://paperswithcode.com/sota/code-documentation-generation-on-2?p=codebert-a-pre-trained-model-for-programming)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codebert-a-pre-trained-model-for-programming/code-documentation-generation-on-5)](https://paperswithcode.com/sota/code-documentation-generation-on-5?p=codebert-a-pre-trained-model-for-programming)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codebert-a-pre-trained-model-for-programming/code-documentation-generation-on-1)](https://paperswithcode.com/sota/code-documentation-generation-on-1?p=codebert-a-pre-trained-model-for-programming)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codebert-a-pre-trained-model-for-programming/code-documentation-generation-on-3)](https://paperswithcode.com/sota/code-documentation-generation-on-3?p=codebert-a-pre-trained-model-for-programming)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codebert-a-pre-trained-model-for-programming/code-documentation-generation-on)](https://paperswithcode.com/sota/code-documentation-generation-on?p=codebert-a-pre-trained-model-for-programming)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codebert-a-pre-trained-model-for-programming/code-documentation-generation-on-4)](https://paperswithcode.com/sota/code-documentation-generation-on-4?p=codebert-a-pre-trained-model-for-programming)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codebert-a-pre-trained-model-for-programming/type-prediction-on-manytypes4typescript)](https://paperswithcode.com/sota/type-prediction-on-manytypes4typescript?p=codebert-a-pre-trained-model-for-programming)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/codebert-a-pre-trained-model-for-programming/code-search-on-codesearchnet)](https://paperswithcode.com/sota/code-search-on-codesearchnet?p=codebert-a-pre-trained-model-for-programming)`

CodeBERT: A Pre-Trained Model for Programming and Natural Languages

Findings of the Association for Computational Linguistics 2020 · Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, Ming Zhou ·

We present CodeBERT, a bimodal pre-trained model for programming language (PL) and nat-ural language (NL). CodeBERT learns general-purpose representations that support downstream NL-PL applications such as natural language codesearch, code documentation generation, etc. We develop CodeBERT with Transformer-based neural architecture, and train it with a hybrid objective function that incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators. This enables us to utilize both bimodal data of NL-PL pairs and unimodal data, where the former provides input tokens for model training while the latter helps to learn better generators. We evaluate CodeBERT on two NL-PL applications by fine-tuning model parameters. Results show that CodeBERT achieves state-of-the-art performance on both natural language code search and code documentation generation tasks. Furthermore, to investigate what type of knowledge is learned in CodeBERT, we construct a dataset for NL-PL probing, and evaluate in a zero-shot setting where parameters of pre-trained models are fixed. Results show that CodeBERT performs better than previous pre-trained models on NL-PL probing.

PDF Abstract Findings of 2020 PDF Findings of 2020 Abstract

Code

Add Remove Mark official

microsoft/CodeBERT official

2,319

salesforce/codet5

2,852

graykode/commit-autosuggestions

385

zhangzwwww/dietcodebert

zhangzwwww/dietcode

See all 9 implementations

Tasks

Add Remove

Code Documentation Generation

Datasets

CodeSearchNet ManyTypes4TypeScript

Results from the Paper

Edit

Ranked #1 on Code Documentation Generation on CodeSearchNet - Go

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Code Documentation Generation	CodeSearchNet	seq2seq	Smoothed BLEU-4	13.36	# 7	Compare
Code Documentation Generation	CodeSearchNet	Transformer	Smoothed BLEU-4	14.31	# 6	Compare
Code Search	CodeSearchNet	CodeBERT	Overall	76.0	# 6	Compare
			Go	69.3	# 6	Compare
			Ruby	70.6	# 6	Compare
			Python	84.0	# 4	Compare
			Java	86.8	# 3	Compare
			JS	74.8	# 3	Compare
			PHP	70.6	# 4	Compare
Code Documentation Generation	CodeSearchNet	CodeBERT (MLM+RTD)	Smoothed BLEU-4	15.99	# 1	Compare
Code Documentation Generation	CodeSearchNet	RoBERTa	Smoothed BLEU-4	14.52	# 5	Compare
Code Documentation Generation	CodeSearchNet	CodeBERT (MLM)	Smoothed BLEU-4	15.55	# 2	Compare
Code Documentation Generation	CodeSearchNet	pre-train w/ code only	Smoothed BLEU-4	15.15	# 3	Compare
Code Documentation Generation	CodeSearchNet	CodeBERT (RTD)	Smoothed BLEU-4	15.03	# 4	Compare
Code Documentation Generation	CodeSearchNet - Go	CodeBERT (RTD)	Smoothed BLEU-4	26.02	# 5	Compare
Code Documentation Generation	CodeSearchNet - Go	RoBERTa	Smoothed BLEU-4	26.09	# 4	Compare
Code Documentation Generation	CodeSearchNet - Go	CodeBERT (MLM)	Smoothed BLEU-4	26.79	# 1	Compare
Code Documentation Generation	CodeSearchNet - Go	seq2seq	Smoothed BLEU-4	23.48	# 6	Compare
Code Documentation Generation	CodeSearchNet - Go	pre-train w/ code only	Smoothed BLEU-4	26.39	# 3	Compare
Code Documentation Generation	CodeSearchNet - Go	CodeBERT (MLM+RTD)	Smoothed BLEU-4	26.66	# 2	Compare
Code Documentation Generation	CodeSearchNet - Java	Transformer	Smoothed BLEU-4	12.57	# 7	Compare
Code Documentation Generation	CodeSearchNet - Java	CodeBERT (MLM+RTD)	Smoothed BLEU-4	14.56	# 2	Compare
Code Documentation Generation	CodeSearchNet - Java	CodeBERT (MLM)	Smoothed BLEU-4	13.59	# 3	Compare
Code Documentation Generation	CodeSearchNet - Java	CodeBERT (RTD)	Smoothed BLEU-4	12.72	# 6	Compare
Code Documentation Generation	CodeSearchNet - Java	pre-train w/ code only	Smoothed BLEU-4	13.07	# 5	Compare
Code Documentation Generation	CodeSearchNet - Java	RoBERTa	Smoothed BLEU-4	13.2	# 4	Compare
Code Documentation Generation	CodeSearchNet - Java	seq2seq	Smoothed BLEU-4	11.42	# 8	Compare
Code Documentation Generation	CodeSearchNet - JavaScript	CodeBERT (MLM)	Smoothed BLEU-4	8.51	# 5	Compare
Code Documentation Generation	CodeSearchNet - JavaScript	CodeBERT (RTD)	Smoothed BLEU-4	8.73	# 4	Compare
Code Documentation Generation	CodeSearchNet - JavaScript	Transformer	Smoothed BLEU-4	25.61	# 1	Compare
Code Documentation Generation	CodeSearchNet - JavaScript	CodeBERT (MLM+RTD)	Smoothed BLEU-4	9.54	# 3	Compare
Code Documentation Generation	CodeSearchNet - JavaScript	RoBERTa	Smoothed BLEU-4	5.72	# 8	Compare
Code Documentation Generation	CodeSearchNet - JavaScript	pre-train w/ code only	Smoothed BLEU-4	8.3	# 6	Compare
Code Documentation Generation	CodeSearchNet - JavaScript	seq2seq	Smoothed BLEU-4	6.88	# 7	Compare
Code Documentation Generation	CodeSearchNet - Php	Transformer	Smoothed BLEU-4	18.25	# 8	Compare
Code Documentation Generation	CodeSearchNet - Php	CodeBERT (MLM)	Smoothed BLEU-4	21	# 3	Compare
Code Documentation Generation	CodeSearchNet - Php	pre-train w/ code only	Smoothed BLEU-4	20.71	# 4	Compare
Code Documentation Generation	CodeSearchNet - Php	RoBERTa	Smoothed BLEU-4	19.9	# 6	Compare
Code Documentation Generation	CodeSearchNet - Php	CodeBERT (MLM+RTD)	Smoothed BLEU-4	21.32	# 2	Compare
Code Documentation Generation	CodeSearchNet - Php	seq2seq	Smoothed BLEU-4	18.4	# 7	Compare
Code Documentation Generation	CodeSearchNet - Php	CodeBERT (RTD)	Smoothed BLEU-4	20.25	# 5	Compare
Code Documentation Generation	CodeSearchNet - Python	pre-train w/ code only	Smoothed BLEU-4	15.12	# 4	Compare
Code Documentation Generation	CodeSearchNet - Python	CodeBERT (MLM+RTD)	Smoothed BLEU-4	15.41	# 3	Compare
Code Documentation Generation	CodeSearchNet - Python	Transformer	Smoothed BLEU-4	13.44	# 6	Compare
Code Documentation Generation	CodeSearchNet - Python	seq2seq	Smoothed BLEU-4	13.04	# 7	Compare
Code Documentation Generation	CodeSearchNet - Python	CodeBERT (MLM)	Smoothed BLEU-4	15.48	# 2	Compare
Code Documentation Generation	CodeSearchNet - Python	RoBERTa	Smoothed BLEU-4	14.92	# 5	Compare
Code Documentation Generation	CodeSearchNet - Ruby	Transformer	Smoothed BLEU-4	7.87	# 4	Compare
Code Documentation Generation	CodeSearchNet - Ruby	pre-train w/ code only	Smoothed BLEU-4	7.36	# 5	Compare
Code Documentation Generation	CodeSearchNet - Ruby	CodeBERT (MLM)	Smoothed BLEU-4	7.95	# 3	Compare
Code Documentation Generation	CodeSearchNet - Ruby	seq2seq	Smoothed BLEU-4	6.96	# 7	Compare
Code Documentation Generation	CodeSearchNet - Ruby	CodeBERT (MLM+RTD)	Smoothed BLEU-4	8.46	# 2	Compare
Code Documentation Generation	CodeSearchNet - Ruby	RoBERTa	Smoothed BLEU-4	7.26	# 6	Compare
Type prediction	ManyTypes4TypeScript	CodeBERT	Average Accuracy	61.72	# 4	Compare
			Average Precision	59.34	# 2	Compare
			Average Recall	59.80	# 2	Compare
			Average F1	59.57	# 2	Compare

Methods

Add Remove

CodeBERT

Edit Social Preview

CodeBERT: A Pre-Trained Model for Programming and Natural Languages

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove