DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
The rapid development of large language models has revolutionized code intelligence in software development. However, the predominance of closed-source models has restricted extensive research and development. To address this, we introduce the DeepSeek-Coder series, a range of open-source code models with sizes from 1.3B to 33B, trained from scratch on 2 trillion tokens. These models are pre-trained on a high-quality project-level code corpus and employ a fill-in-the-blank task with a 16K window to enhance code generation and infilling. Our extensive evaluations demonstrate that DeepSeek-Coder not only achieves state-of-the-art performance among open-source code models across multiple benchmarks but also surpasses existing closed-source models like Codex and GPT-3.5. Furthermore, DeepSeek-Coder models are under a permissive license that allows for both research and unrestricted commercial use.
PDF AbstractCode
Spaces
MMLU
GSM8K
HumanEval
HellaSwag
MATH
SVAMP
DS-1000
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Benchmark |
|---|---|---|---|---|---|---|
| Code Generation | HumanEval | GPT-4 (0-shot) | Pass@1 | 76.5 | # 13 | |
| Code Generation | HumanEval | GPT-3.5 Turbo (zero-shot) | Pass@1 | 64.9 | # 30 | |
| Code Generation | HumanEval | DeepSeek-Coder-Instruct 33B (0-shot) | Pass@1 | 69.2 | # 23 | |
| Code Generation | HumanEval | DeepSeek-Coder-Base 1.3B (zero-shot) | Pass@1 | 28.3 | # 85 | |
| Code Generation | HumanEval | DeepSeek-Coder-Base 6.7B (zero-shot) | Pass@1 | 44.7 | # 53 | |
| Code Generation | HumanEval | DeepSeek-Coder-Instruct 6.7B (0-shot) | Pass@1 | 66.1 | # 27 | |
| Code Generation | HumanEval | DeepSeek-Coder-Instruct 1.3B (zero-shot) | Pass@1 | 48.4 | # 46 | |
| Code Generation | HumanEval | DeepSeek-Coder-Base 33B (zero-shot) | Pass@1 | 50.3 | # 43 | |
| Code Generation | MBPP | DeepSeek-Coder-Base 6.7B (few-shot) | Accuracy | 60.6 | # 36 | |
| Code Generation | MBPP | DeepSeek-Coder-Instruct 6.7B (few-shot) | Accuracy | 65.4 | # 28 | |
| Code Generation | MBPP | DeepSeek-Coder-Base 33B (few-shot) | Accuracy | 66 | # 26 | |
| Code Generation | MBPP | GPT-3.5 Turbo (few-shot) | Accuracy | 70.8 | # 16 | |
| Code Generation | MBPP | GPT-4 (few-shot) | Accuracy | 80 | # 11 | |
| Code Generation | MBPP | DeepSeek-Coder-Instruct 1.3B (few-shot) | Accuracy | 49.4 | # 48 | |
| Code Generation | MBPP | DeepSeek-Coder-Base 1.3B (few-shot) | Accuracy | 46.2 | # 60 | |
| Code Generation | MBPP | DeepSeek-Coder-Instruct 33B (few-shot) | Accuracy | 70 | # 18 |
