CodeTrans: Towards Cracking the Language of Silicone's Code Through Self-Supervised Deep Learning and High Performance Computing

Currently, a growing number of mature natural language processing applications make people's life more convenient. Such applications are built by source code - the language in software engineering... (read more)

PDF Abstract

Datasets


TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK BENCHMARK
Program Synthesis AlgoLisp CodeTrans-MT-TF-Small Accuracy 90.31 # 1
Code Documentation Generation CodeSearchNet - Go CodeTrans-TF-Large Smoothed BLEU-4 19.54 # 1
Code Documentation Generation CodeSearchNet - Java CodeTrans-MT-Large Smoothed BLEU-4 21.87 # 1
Code Documentation Generation CodeSearchNet - JavaScript CodeTrans-TF-Large Smoothed BLEU-4 18.98 # 1
Code Documentation Generation CodeSearchNet - Php CodeTrans-MT-Base Smoothed BLEU-4 26.23 # 1
Code Documentation Generation CodeSearchNet - Python CodeTrans-MT-Base Smoothed BLEU-4 20.39 # 1
Code Documentation Generation CodeSearchNet - Ruby CodeTrans-MT-Base Smoothed BLEU-4 15.26 # 1
Git Commit Message Generation CommitGen CodeTrans-TF-Large BLEU-4 44.41 # 1
API Sequence Recommendation DeepAPI CodeTrans-MT-TF-Large BLEU-4 73.39 # 1
Code Comment Generation DeepCom CodeTrans-TF-Large Smoothed BLEU-4 39.50 # 1
Source Code Summarization Summarizing Source Code using a Neural Attention Model - C# CodeTrans-MT-Large Smoothed BLEU-4 23.57 # 1
Source Code Summarization Summarizing Source Code using a Neural Attention Model - Python CodeTrans-MT-Base Smoothed BLEU-4 13.37 # 1
Source Code Summarization Summarizing Source Code using a Neural Attention Model - SQL CodeTrans-MT-TF-Large Smoothed BLEU-4 19.98 # 1

Methods used in the Paper


METHOD TYPE
BPE
Subword Segmentation
GLU
Activation Functions
GELU
Activation Functions
Inverse Square Root Schedule
Learning Rate Schedules
Adafactor
Stochastic Optimization
Dense Connections
Feedforward Networks
Softmax
Output Functions
Attention Dropout
Regularization
Dropout
Regularization
Residual Connection
Skip Connections
Layer Normalization
Normalization
Scaled Dot-Product Attention
Attention Mechanisms
Multi-Head Attention
Attention Modules
SentencePiece
Tokenizers
T5
Transformers